2024-03-12T03:03:14.140 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-12T03:03:14.140 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591867 --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} --name teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi --verbose --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591867/orig.config.yaml 2024-03-12T03:03:14.169 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-12T03:03:14.255 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591867 branch: main description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591867' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c owner: scheduled_teuthology@teuthology priority: 200 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 6815 sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b sleep_before_teardown: 0 subset: 10/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFYNbtVB+BohaOZTEaOFaDawSnUznaOZ5FN7PioXQAXVoeMucFugO3WyrhyDpdNtczqNSnh6S9oKMJL37ThbJtc= smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAYFk7k/QB/PoM6AOmNI5494y5ayWCH6aX0cdUaBq2+RJAuaTowVDAAVHFaLcnk6YosM0BHs0S+TawzsiLj2a+A= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJtZCqsDfYX2qkp536edJjx3PIPmUR00AqgCzTDB0NPMoQW+7MEeN5bizgrJQ6inGZm6myvFRPSHQlTS5nExkb0= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_20:08:13 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-12T03:03:14.255 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa; will attempt to use it 2024-03-12T03:03:14.256 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks 2024-03-12T03:03:14.256 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-12T03:03:14.257 INFO:teuthology.task.internal:Checking packages... 2024-03-12T03:03:14.282 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '2b0101eb74d60610d544cb8a473e23a35ac1cc0b' 2024-03-12T03:03:14.282 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-12T03:03:14.282 INFO:teuthology.packaging:ref: None 2024-03-12T03:03:14.283 INFO:teuthology.packaging:tag: None 2024-03-12T03:03:14.283 INFO:teuthology.packaging:branch: main 2024-03-12T03:03:14.283 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:03:14.283 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-12T03:03:14.516 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1998.g1c349451 2024-03-12T03:03:14.517 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-12T03:03:14.525 INFO:teuthology.task.internal:no buildpackages task found 2024-03-12T03:03:14.525 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-12T03:03:14.532 INFO:teuthology.task.internal:Saving configuration 2024-03-12T03:03:14.544 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-12T03:03:14.550 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T03:03:14.574 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591867', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:58:09.461162', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFYNbtVB+BohaOZTEaOFaDawSnUznaOZ5FN7PioXQAXVoeMucFugO3WyrhyDpdNtczqNSnh6S9oKMJL37ThbJtc='} 2024-03-12T03:03:14.601 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591867', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:58:09.459344', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAYFk7k/QB/PoM6AOmNI5494y5ayWCH6aX0cdUaBq2+RJAuaTowVDAAVHFaLcnk6YosM0BHs0S+TawzsiLj2a+A='} 2024-03-12T03:03:14.622 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591867', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:58:09.462296', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJtZCqsDfYX2qkp536edJjx3PIPmUR00AqgCzTDB0NPMoQW+7MEeN5bizgrJQ6inGZm6myvFRPSHQlTS5nExkb0='} 2024-03-12T03:03:14.623 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-12T03:03:14.628 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-12T03:03:14.629 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-12T03:03:14.629 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-12T03:03:14.629 INFO:teuthology.run_tasks:Running task console_log... 2024-03-12T03:03:14.733 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f13eb680f70>, signals=[15]) 2024-03-12T03:03:14.733 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-12T03:03:14.739 INFO:teuthology.task.internal:Opening connections... 2024-03-12T03:03:14.739 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2024-03-12T03:03:14.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:03:14.852 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2024-03-12T03:03:14.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:03:14.956 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2024-03-12T03:03:14.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:03:15.046 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-12T03:03:15.055 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2024-03-12T03:03:15.073 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2024-03-12T03:03:15.073 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2024-03-12T03:03:15.135 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream" 2024-03-12T03:03:15.135 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="9" 2024-03-12T03:03:15.135 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2024-03-12T03:03:15.136 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2024-03-12T03:03:15.136 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="9" 2024-03-12T03:03:15.136 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el9" 2024-03-12T03:03:15.136 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T03:03:15.136 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2024-03-12T03:03:15.136 INFO:teuthology.orchestra.run.smithi007.stdout:LOGO="fedora-logo-icon" 2024-03-12T03:03:15.136 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T03:03:15.136 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/" 2024-03-12T03:03:15.136 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T03:03:15.136 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T03:03:15.136 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T03:03:15.137 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-03-12T03:03:15.157 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2024-03-12T03:03:15.176 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2024-03-12T03:03:15.176 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2024-03-12T03:03:15.235 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Stream" 2024-03-12T03:03:15.235 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="9" 2024-03-12T03:03:15.235 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos" 2024-03-12T03:03:15.235 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora" 2024-03-12T03:03:15.235 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="9" 2024-03-12T03:03:15.235 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el9" 2024-03-12T03:03:15.235 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T03:03:15.235 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31" 2024-03-12T03:03:15.235 INFO:teuthology.orchestra.run.smithi033.stdout:LOGO="fedora-logo-icon" 2024-03-12T03:03:15.235 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T03:03:15.236 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://centos.org/" 2024-03-12T03:03:15.236 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T03:03:15.236 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T03:03:15.236 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T03:03:15.236 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2024-03-12T03:03:15.257 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2024-03-12T03:03:15.275 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2024-03-12T03:03:15.276 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2024-03-12T03:03:15.337 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2024-03-12T03:03:15.337 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="9" 2024-03-12T03:03:15.337 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2024-03-12T03:03:15.337 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2024-03-12T03:03:15.337 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="9" 2024-03-12T03:03:15.337 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el9" 2024-03-12T03:03:15.337 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T03:03:15.337 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2024-03-12T03:03:15.337 INFO:teuthology.orchestra.run.smithi170.stdout:LOGO="fedora-logo-icon" 2024-03-12T03:03:15.337 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T03:03:15.338 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2024-03-12T03:03:15.338 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T03:03:15.338 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T03:03:15.338 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T03:03:15.338 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2024-03-12T03:03:15.358 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-12T03:03:15.367 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-12T03:03:15.513 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-12T03:03:15.513 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2024-03-12T03:03:15.516 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2024-03-12T03:03:15.518 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2024-03-12T03:03:15.534 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-12T03:03:15.540 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-12T03:03:15.541 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2024-03-12T03:03:15.573 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2024-03-12T03:03:15.576 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2024-03-12T03:03:15.658 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-12T03:03:15.742 INFO:teuthology.run_tasks:Running task kernel... 2024-03-12T03:03:15.758 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-12T03:03:15.758 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-12T03:03:15.759 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T03:03:15.759 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T03:03:15.760 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T03:03:15.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:03:15.776 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-03-12T03:03:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:03:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:03:15.778 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-03-12T03:03:15.778 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-03-12T03:03:15.835 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-361.el9.x86_64 2024-03-12T03:03:15.835 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.14.0-361.el9.x86_64 2024-03-12T03:03:15.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2024-03-12T03:03:15.837 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-361.el9.x86_64 2024-03-12T03:03:15.837 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-361.el9.x86_64 2024-03-12T03:03:15.837 INFO:teuthology.task.kernel:Running kernel on smithi170: 5.14.0-361.el9.x86_64 2024-03-12T03:03:15.838 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum install -y kernel 2024-03-12T03:03:15.838 INFO:teuthology.task.kernel:Running kernel on smithi033: 5.14.0-361.el9.x86_64 2024-03-12T03:03:15.838 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum install -y kernel 2024-03-12T03:03:17.861 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-12T03:03:18.035 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-12T03:03:18.100 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-12T03:03:19.303 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-12T03:03:19.535 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-12T03:03:19.559 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-12T03:03:20.594 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-12T03:03:22.567 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-12T03:03:23.085 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-12T03:03:26.753 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 7.3 MB/s | 19 MB 00:02 2024-03-12T03:03:29.874 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 823 kB/s | 8.0 MB 00:09 2024-03-12T03:03:35.183 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 706 kB/s | 6.1 MB 00:08 2024-03-12T03:03:36.793 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 15 kB 00:00 2024-03-12T03:03:38.136 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-12T03:03:38.961 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 720 kB/s | 6.1 MB 00:08 2024-03-12T03:03:40.528 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 15 kB 00:00 2024-03-12T03:03:42.471 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-12T03:03:43.277 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-12T03:03:45.575 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T03:03:45.576 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T03:03:45.667 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-12T03:03:45.681 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-12T03:03:45.681 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-03-12T03:03:45.681 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-12T03:03:45.681 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-12T03:03:45.682 INFO:teuthology.orchestra.run.smithi033.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T03:03:45.682 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-12T03:03:45.682 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T03:03:45.682 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T03:03:45.682 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T03:03:45.682 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-12T03:03:45.682 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-12T03:03:45.682 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-12T03:03:45.683 INFO:teuthology.orchestra.run.smithi033.stdout:Install 4 Packages 2024-03-12T03:03:45.683 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-12T03:03:45.685 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 100 M 2024-03-12T03:03:45.686 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 123 M 2024-03-12T03:03:45.686 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-12T03:03:47.387 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-03-12T03:03:49.511 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T03:03:49.511 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T03:03:49.600 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-12T03:03:49.608 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-12T03:03:49.608 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-03-12T03:03:49.608 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-12T03:03:49.608 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-12T03:03:49.608 INFO:teuthology.orchestra.run.smithi170.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T03:03:49.609 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-12T03:03:49.609 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T03:03:49.609 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T03:03:49.609 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T03:03:49.609 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-12T03:03:49.609 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-12T03:03:49.609 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-12T03:03:49.609 INFO:teuthology.orchestra.run.smithi170.stdout:Install 4 Packages 2024-03-12T03:03:49.609 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-12T03:03:49.610 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 100 M 2024-03-12T03:03:49.610 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 123 M 2024-03-12T03:03:49.610 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-12T03:03:50.136 INFO:teuthology.orchestra.run.smithi033.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 1.4 MB/s | 6.0 MB 00:04 2024-03-12T03:03:52.120 INFO:teuthology.orchestra.run.smithi033.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 3.4 MB/s | 21 MB 00:06 2024-03-12T03:03:53.586 INFO:teuthology.orchestra.run.smithi033.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 5.2 MB/s | 39 MB 00:07 2024-03-12T03:03:55.071 INFO:teuthology.orchestra.run.smithi170.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 1.1 MB/s | 6.0 MB 00:05 2024-03-12T03:03:56.586 INFO:teuthology.orchestra.run.smithi033.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 5.2 MB/s | 33 MB 00:06 2024-03-12T03:03:56.586 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:03:56.586 INFO:teuthology.orchestra.run.smithi033.stdout:Total 9.1 MB/s | 100 MB 00:10 2024-03-12T03:03:57.511 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-12T03:03:57.971 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-12T03:03:57.971 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-12T03:03:58.130 INFO:teuthology.orchestra.run.smithi170.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 2.5 MB/s | 21 MB 00:08 2024-03-12T03:03:59.394 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-12T03:03:59.395 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-12T03:04:00.363 INFO:teuthology.orchestra.run.smithi170.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 3.7 MB/s | 39 MB 00:10 2024-03-12T03:04:00.680 INFO:teuthology.orchestra.run.smithi170.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 6.0 MB/s | 33 MB 00:05 2024-03-12T03:04:00.680 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:04:00.680 INFO:teuthology.orchestra.run.smithi170.stdout:Total 9.0 MB/s | 100 MB 00:11 2024-03-12T03:04:01.564 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-12T03:04:01.929 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-12T03:04:01.930 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-12T03:04:02.263 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-12T03:04:02.952 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:04:03.078 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:04:03.298 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-12T03:04:03.299 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-12T03:04:05.988 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:04:06.010 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-12T03:04:06.172 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:04:06.660 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:04:06.797 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:04:09.571 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:04:09.752 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:04:10.537 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 488 kB/s | 19 MB 00:39 2024-03-12T03:04:11.302 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:04:11.426 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:04:14.484 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:04:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:04:16.585 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:04:19.421 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:04:22.711 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 734 kB/s | 6.1 MB 00:08 2024-03-12T03:04:24.137 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 15 kB 00:00 2024-03-12T03:04:47.521 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:04:47.521 INFO:teuthology.orchestra.run.smithi033.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T03:04:47.521 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-12T03:04:47.609 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:04:48.081 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:04:48.081 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:04:48.081 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:04:48.081 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:04:48.456 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:04:48.456 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-12T03:04:48.457 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-12T03:04:48.457 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T03:04:48.457 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T03:04:48.457 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T03:04:48.457 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T03:04:48.457 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-12T03:04:48.457 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-12T03:04:48.713 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:04:48.713 INFO:teuthology.orchestra.run.smithi170.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T03:04:48.713 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-12T03:04:48.801 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:04:49.034 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:04:49.175 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:04:49.176 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:04:49.176 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T03:04:49.176 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T03:04:49.176 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T03:04:49.176 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-12T03:04:49.176 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T03:04:49.176 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:04:49.253 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:04:49.253 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:04:49.253 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:04:49.254 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:04:49.302 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:04:49.302 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:04:49.303 DEBUG:teuthology.orchestra.run.smithi033:> sudo rpm -qi grub2-tools 2024-03-12T03:04:49.340 INFO:teuthology.orchestra.run.smithi033.stdout:Name : grub2-tools 2024-03-12T03:04:49.340 INFO:teuthology.orchestra.run.smithi033.stdout:Epoch : 1 2024-03-12T03:04:49.340 INFO:teuthology.orchestra.run.smithi033.stdout:Version : 2.06 2024-03-12T03:04:49.341 INFO:teuthology.orchestra.run.smithi033.stdout:Release : 68.el9 2024-03-12T03:04:49.341 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: x86_64 2024-03-12T03:04:49.341 INFO:teuthology.orchestra.run.smithi033.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T03:04:49.341 INFO:teuthology.orchestra.run.smithi033.stdout:Group : Unspecified 2024-03-12T03:04:49.341 INFO:teuthology.orchestra.run.smithi033.stdout:Size : 8264254 2024-03-12T03:04:49.341 INFO:teuthology.orchestra.run.smithi033.stdout:License : GPLv3+ 2024-03-12T03:04:49.341 INFO:teuthology.orchestra.run.smithi033.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T03:04:49.341 INFO:teuthology.orchestra.run.smithi033.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T03:04:49.341 INFO:teuthology.orchestra.run.smithi033.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T03:04:49.341 INFO:teuthology.orchestra.run.smithi033.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T03:04:49.341 INFO:teuthology.orchestra.run.smithi033.stdout:Packager : builder@centos.org 2024-03-12T03:04:49.341 INFO:teuthology.orchestra.run.smithi033.stdout:Vendor : CentOS 2024-03-12T03:04:49.342 INFO:teuthology.orchestra.run.smithi033.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T03:04:49.342 INFO:teuthology.orchestra.run.smithi033.stdout:Summary : Support tools for GRUB. 2024-03-12T03:04:49.342 INFO:teuthology.orchestra.run.smithi033.stdout:Description : 2024-03-12T03:04:49.342 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-12T03:04:49.342 INFO:teuthology.orchestra.run.smithi033.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T03:04:49.342 INFO:teuthology.orchestra.run.smithi033.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T03:04:49.342 INFO:teuthology.orchestra.run.smithi033.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T03:04:49.342 INFO:teuthology.orchestra.run.smithi033.stdout:hardware devices. 2024-03-12T03:04:49.342 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-12T03:04:49.342 INFO:teuthology.orchestra.run.smithi033.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T03:04:49.343 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T03:04:49.343 INFO:teuthology.task.kernel:Updating grub on smithi033 to boot 5.14.0-427.el9.x86_64 2024-03-12T03:04:49.343 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T03:04:49.668 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:04:49.668 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-12T03:04:49.668 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-12T03:04:49.668 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T03:04:49.668 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T03:04:49.668 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T03:04:49.668 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T03:04:49.668 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-12T03:04:49.668 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-12T03:04:49.959 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2024-03-12T03:04:50.186 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:04:50.324 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:04:50.324 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:04:50.325 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T03:04:50.325 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T03:04:50.325 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T03:04:50.325 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-12T03:04:50.325 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T03:04:50.325 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:04:50.446 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:04:50.446 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:04:50.446 DEBUG:teuthology.orchestra.run.smithi170:> sudo rpm -qi grub2-tools 2024-03-12T03:04:50.480 INFO:teuthology.orchestra.run.smithi170.stdout:Name : grub2-tools 2024-03-12T03:04:50.480 INFO:teuthology.orchestra.run.smithi170.stdout:Epoch : 1 2024-03-12T03:04:50.480 INFO:teuthology.orchestra.run.smithi170.stdout:Version : 2.06 2024-03-12T03:04:50.480 INFO:teuthology.orchestra.run.smithi170.stdout:Release : 68.el9 2024-03-12T03:04:50.480 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: x86_64 2024-03-12T03:04:50.481 INFO:teuthology.orchestra.run.smithi170.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T03:04:50.481 INFO:teuthology.orchestra.run.smithi170.stdout:Group : Unspecified 2024-03-12T03:04:50.481 INFO:teuthology.orchestra.run.smithi170.stdout:Size : 8264254 2024-03-12T03:04:50.481 INFO:teuthology.orchestra.run.smithi170.stdout:License : GPLv3+ 2024-03-12T03:04:50.481 INFO:teuthology.orchestra.run.smithi170.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T03:04:50.481 INFO:teuthology.orchestra.run.smithi170.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T03:04:50.481 INFO:teuthology.orchestra.run.smithi170.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T03:04:50.481 INFO:teuthology.orchestra.run.smithi170.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T03:04:50.481 INFO:teuthology.orchestra.run.smithi170.stdout:Packager : builder@centos.org 2024-03-12T03:04:50.481 INFO:teuthology.orchestra.run.smithi170.stdout:Vendor : CentOS 2024-03-12T03:04:50.481 INFO:teuthology.orchestra.run.smithi170.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T03:04:50.481 INFO:teuthology.orchestra.run.smithi170.stdout:Summary : Support tools for GRUB. 2024-03-12T03:04:50.481 INFO:teuthology.orchestra.run.smithi170.stdout:Description : 2024-03-12T03:04:50.481 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-12T03:04:50.482 INFO:teuthology.orchestra.run.smithi170.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T03:04:50.482 INFO:teuthology.orchestra.run.smithi170.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T03:04:50.482 INFO:teuthology.orchestra.run.smithi170.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T03:04:50.482 INFO:teuthology.orchestra.run.smithi170.stdout:hardware devices. 2024-03-12T03:04:50.482 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-12T03:04:50.482 INFO:teuthology.orchestra.run.smithi170.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T03:04:50.483 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T03:04:50.483 INFO:teuthology.task.kernel:Updating grub on smithi170 to boot 5.14.0-427.el9.x86_64 2024-03-12T03:04:50.483 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T03:04:51.119 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2024-03-12T03:04:51.373 INFO:teuthology.orchestra.run.smithi033.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T03:04:51.411 INFO:teuthology.orchestra.run.smithi033.stderr:done 2024-03-12T03:04:51.413 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2024-03-12T03:04:51.429 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.d183mKDYuN 2024-03-12T03:04:51.429 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.d183mKDYuN 2024-03-12T03:04:51.500 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.d183mKDYuN 2024-03-12T03:04:51.645 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.d183mKDYuN is 6KB 2024-03-12T03:04:51.695 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.d183mKDYuN 2024-03-12T03:04:51.709 DEBUG:teuthology.orchestra.run.smithi033:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T03:04:51.786 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T03:04:51.786 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T03:04:51.786 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T03:04:51.786 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T03:04:51.788 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T03:04:51.952 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2024-03-12T03:04:52.452 INFO:teuthology.orchestra.run.smithi170.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T03:04:52.496 INFO:teuthology.orchestra.run.smithi170.stderr:done 2024-03-12T03:04:52.499 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2024-03-12T03:04:52.514 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.YbGjdFttHf 2024-03-12T03:04:52.514 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YbGjdFttHf 2024-03-12T03:04:52.582 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.YbGjdFttHf 2024-03-12T03:04:52.717 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.YbGjdFttHf is 6KB 2024-03-12T03:04:52.766 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.YbGjdFttHf 2024-03-12T03:04:52.780 DEBUG:teuthology.orchestra.run.smithi170:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T03:04:52.844 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T03:04:52.844 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T03:04:52.844 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T03:04:52.844 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T03:04:52.845 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T03:04:52.999 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2024-03-12T03:05:21.955 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-12T03:05:21.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-12T03:05:21.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:05:23.002 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-12T03:05:23.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-12T03:05:23.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:05:40.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-03-12T03:05:41.410 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2024-03-12T03:05:49.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-12T03:05:49.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:05:50.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-12T03:05:50.412 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:05:53.470 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2024-03-12T03:06:05.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-12T03:06:05.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:06:21.236 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-03-12T03:06:21.649 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-12T03:06:21.649 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-12T03:06:21.650 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-03-12T03:06:21.664 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-427.el9.x86_64 2024-03-12T03:06:21.664 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T03:06:21.664 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T03:06:21.664 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T03:06:22.665 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-12T03:06:22.666 DEBUG:teuthology.orchestra.run.smithi033:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T03:06:22.746 INFO:teuthology.orchestra.run.smithi033.stdout:ttyS1 2024-03-12T03:06:22.795 DEBUG:teuthology.parallel:result is None 2024-03-12T03:06:37.379 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-03-12T03:06:37.800 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-12T03:06:37.800 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-427.el9.x86_64"... 2024-03-12T03:06:37.800 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-03-12T03:06:37.815 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-427.el9.x86_64 2024-03-12T03:06:37.815 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T03:06:37.815 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T03:06:37.815 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T03:06:38.816 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-12T03:06:38.816 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T03:06:38.893 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2024-03-12T03:06:38.934 DEBUG:teuthology.parallel:result is None 2024-03-12T03:07:09.990 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 127 kB/s | 21 MB 02:45 2024-03-12T03:07:15.056 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-12T03:07:17.338 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T03:07:17.338 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T03:07:17.436 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-12T03:07:17.452 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-12T03:07:17.452 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-03-12T03:07:17.452 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-12T03:07:17.453 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-12T03:07:17.453 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T03:07:17.453 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-12T03:07:17.453 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T03:07:17.453 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T03:07:17.453 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T03:07:17.453 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:07:17.453 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-12T03:07:17.453 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-12T03:07:17.453 INFO:teuthology.orchestra.run.smithi007.stdout:Install 4 Packages 2024-03-12T03:07:17.453 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:07:17.453 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 100 M 2024-03-12T03:07:17.453 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 123 M 2024-03-12T03:07:17.453 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-12T03:07:17.991 INFO:teuthology.orchestra.run.smithi007.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 14 MB/s | 6.0 MB 00:00 2024-03-12T03:07:18.941 INFO:teuthology.orchestra.run.smithi007.stdout:(2/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 29 MB/s | 39 MB 00:01 2024-03-12T03:07:19.217 INFO:teuthology.orchestra.run.smithi007.stdout:(3/4): kernel-core-5.14.0-427.el9.x86_64.rpm 13 MB/s | 21 MB 00:01 2024-03-12T03:07:19.642 INFO:teuthology.orchestra.run.smithi007.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 20 MB/s | 33 MB 00:01 2024-03-12T03:07:19.642 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:07:19.642 INFO:teuthology.orchestra.run.smithi007.stdout:Total 45 MB/s | 100 MB 00:02 2024-03-12T03:07:20.563 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-12T03:07:21.005 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-12T03:07:21.005 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-12T03:07:22.442 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-12T03:07:22.443 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-12T03:07:25.303 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-12T03:07:25.968 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:07:26.103 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:07:29.059 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:07:29.240 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:07:34.367 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:07:34.485 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:07:39.652 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:08:10.511 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:08:10.511 INFO:teuthology.orchestra.run.smithi007.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T03:08:10.511 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:08:10.599 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:08:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:08:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:08:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:08:11.102 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:08:11.568 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:08:11.569 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:08:11.569 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-12T03:08:11.569 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T03:08:11.569 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T03:08:11.569 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T03:08:11.569 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T03:08:11.569 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:08:11.569 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-12T03:08:12.099 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:08:12.251 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:08:12.252 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:08:12.252 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T03:08:12.252 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T03:08:12.252 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T03:08:12.252 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-12T03:08:12.252 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T03:08:12.252 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:08:12.379 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:08:12.380 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:08:12.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -qi grub2-tools 2024-03-12T03:08:12.426 INFO:teuthology.orchestra.run.smithi007.stdout:Name : grub2-tools 2024-03-12T03:08:12.426 INFO:teuthology.orchestra.run.smithi007.stdout:Epoch : 1 2024-03-12T03:08:12.426 INFO:teuthology.orchestra.run.smithi007.stdout:Version : 2.06 2024-03-12T03:08:12.427 INFO:teuthology.orchestra.run.smithi007.stdout:Release : 68.el9 2024-03-12T03:08:12.427 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: x86_64 2024-03-12T03:08:12.427 INFO:teuthology.orchestra.run.smithi007.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T03:08:12.427 INFO:teuthology.orchestra.run.smithi007.stdout:Group : Unspecified 2024-03-12T03:08:12.427 INFO:teuthology.orchestra.run.smithi007.stdout:Size : 8264254 2024-03-12T03:08:12.427 INFO:teuthology.orchestra.run.smithi007.stdout:License : GPLv3+ 2024-03-12T03:08:12.427 INFO:teuthology.orchestra.run.smithi007.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T03:08:12.427 INFO:teuthology.orchestra.run.smithi007.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T03:08:12.427 INFO:teuthology.orchestra.run.smithi007.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T03:08:12.427 INFO:teuthology.orchestra.run.smithi007.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T03:08:12.427 INFO:teuthology.orchestra.run.smithi007.stdout:Packager : builder@centos.org 2024-03-12T03:08:12.427 INFO:teuthology.orchestra.run.smithi007.stdout:Vendor : CentOS 2024-03-12T03:08:12.427 INFO:teuthology.orchestra.run.smithi007.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T03:08:12.428 INFO:teuthology.orchestra.run.smithi007.stdout:Summary : Support tools for GRUB. 2024-03-12T03:08:12.428 INFO:teuthology.orchestra.run.smithi007.stdout:Description : 2024-03-12T03:08:12.428 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:08:12.428 INFO:teuthology.orchestra.run.smithi007.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T03:08:12.428 INFO:teuthology.orchestra.run.smithi007.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T03:08:12.428 INFO:teuthology.orchestra.run.smithi007.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T03:08:12.428 INFO:teuthology.orchestra.run.smithi007.stdout:hardware devices. 2024-03-12T03:08:12.428 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:08:12.428 INFO:teuthology.orchestra.run.smithi007.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T03:08:12.429 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T03:08:12.429 INFO:teuthology.task.kernel:Updating grub on smithi007 to boot 5.14.0-427.el9.x86_64 2024-03-12T03:08:12.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T03:08:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2024-03-12T03:08:14.462 INFO:teuthology.orchestra.run.smithi007.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T03:08:14.494 INFO:teuthology.orchestra.run.smithi007.stderr:done 2024-03-12T03:08:14.496 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-03-12T03:08:14.511 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.jRgZJHTq4i 2024-03-12T03:08:14.511 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jRgZJHTq4i 2024-03-12T03:08:14.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.jRgZJHTq4i 2024-03-12T03:08:14.734 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.jRgZJHTq4i is 6KB 2024-03-12T03:08:14.784 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.jRgZJHTq4i 2024-03-12T03:08:14.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T03:08:14.868 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T03:08:14.868 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T03:08:14.868 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T03:08:14.868 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T03:08:14.870 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T03:08:15.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2024-03-12T03:08:45.033 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-12T03:08:45.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-12T03:08:45.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:09:03.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2024-03-12T03:09:12.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-12T03:09:12.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:10:12.395 DEBUG:teuthology.orchestra.remote:timed out 2024-03-12T03:10:24.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-12T03:10:24.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:10:24.682 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-03-12T03:10:25.119 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-12T03:10:25.119 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-12T03:10:25.120 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-03-12T03:10:25.135 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-427.el9.x86_64 2024-03-12T03:10:25.135 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T03:10:25.135 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T03:10:25.135 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T03:10:26.136 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-12T03:10:26.136 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T03:10:26.206 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2024-03-12T03:10:26.238 DEBUG:teuthology.parallel:result is None 2024-03-12T03:10:26.238 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-12T03:10:26.248 INFO:teuthology.task.internal:Creating test directory... 2024-03-12T03:10:26.248 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T03:10:26.250 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T03:10:26.253 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T03:10:26.271 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-12T03:10:26.277 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-12T03:10:26.287 INFO:teuthology.task.internal:Creating archive directory... 2024-03-12T03:10:26.287 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T03:10:26.307 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T03:10:26.310 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T03:10:26.343 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-12T03:10:26.349 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-12T03:10:26.349 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T03:10:26.369 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T03:10:26.374 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T03:10:26.403 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:10:26.412 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:10:26.414 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:10:26.417 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:10:26.427 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:10:26.428 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:10:26.430 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-12T03:10:26.438 INFO:teuthology.task.internal:Configuring sudo... 2024-03-12T03:10:26.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T03:10:26.461 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T03:10:26.472 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T03:10:26.501 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-12T03:10:26.510 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-12T03:10:26.510 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T03:10:26.532 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T03:10:26.542 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T03:10:26.557 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:10:26.628 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:10:26.684 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:10:26.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:10:26.811 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-12T03:10:26.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T03:10:26.881 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:10:26.923 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:10:26.981 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:10:27.064 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:10:27.108 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-12T03:10:27.108 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T03:10:27.176 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:10:27.221 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:10:27.278 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:10:27.362 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:10:27.405 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-12T03:10:27.406 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T03:10:27.475 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2024-03-12T03:10:27.477 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2024-03-12T03:10:27.480 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2024-03-12T03:10:27.518 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T03:10:27.530 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T03:10:27.571 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T03:10:27.898 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-12T03:10:27.905 INFO:teuthology.task.internal:Starting timer... 2024-03-12T03:10:27.906 INFO:teuthology.run_tasks:Running task pcp... 2024-03-12T03:10:27.915 INFO:teuthology.run_tasks:Running task selinux... 2024-03-12T03:10:27.924 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-12T03:10:27.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2024-03-12T03:10:27.989 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2024-03-12T03:10:27.991 DEBUG:teuthology.orchestra.run.smithi033:> sudo service auditd rotate 2024-03-12T03:10:28.046 INFO:teuthology.orchestra.run.smithi033.stdout:Rotating logs: 2024-03-12T03:10:28.048 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2024-03-12T03:10:28.109 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2024-03-12T03:10:28.110 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-12T03:10:28.110 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2024-03-12T03:10:28.133 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2024-03-12T03:10:28.134 DEBUG:teuthology.orchestra.run.smithi033:> /usr/sbin/getenforce 2024-03-12T03:10:28.158 INFO:teuthology.orchestra.run.smithi033.stdout:Permissive 2024-03-12T03:10:28.158 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2024-03-12T03:10:28.184 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2024-03-12T03:10:28.185 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi033.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi170.front.sepia.ceph.com': 'permissive'} 2024-03-12T03:10:28.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T03:10:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:10:28.215 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T03:10:28.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:10:28.244 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-12T03:10:28.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:10:28.272 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-12T03:10:28.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2024-03-12T03:10:28.310 DEBUG:teuthology.orchestra.run.smithi033:> sudo /usr/sbin/setenforce permissive 2024-03-12T03:10:28.351 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2024-03-12T03:10:28.387 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-12T03:10:28.399 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-12T03:10:28.416 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-12T03:10:28.417 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi033.front.sepia.ceph.com,smithi170.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-12T03:16:53.082 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi033.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com')] 2024-03-12T03:16:53.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-12T03:16:53.085 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:16:53.178 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-03-12T03:16:53.265 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-12T03:16:53.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-12T03:16:53.266 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:16:53.359 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-03-12T03:16:53.434 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-12T03:16:53.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-12T03:16:53.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:16:53.523 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-03-12T03:16:53.597 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-12T03:16:53.598 INFO:teuthology.run_tasks:Running task clock... 2024-03-12T03:16:53.611 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-12T03:16:53.612 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T03:16:53.612 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T03:16:53.614 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T03:16:53.615 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T03:16:53.617 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T03:16:53.617 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T03:16:53.648 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T03:16:53.649 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T03:16:53.666 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T03:16:53.668 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T03:16:53.669 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T03:16:53.686 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T03:16:53.697 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2024-03-12T03:16:53.699 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: ntpd: command not found 2024-03-12T03:16:53.711 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2024-03-12T03:16:53.714 INFO:teuthology.orchestra.run.smithi033.stdout:506 Cannot talk to daemon 2024-03-12T03:16:53.716 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2024-03-12T03:16:53.728 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T03:16:53.729 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2024-03-12T03:16:53.730 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T03:16:53.745 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T03:16:53.745 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T03:16:53.747 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T03:16:53.762 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T03:16:53.797 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-03-12T03:16:53.800 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:16:53.804 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T03:16:53.805 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-03-12T03:16:53.805 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:16:53.805 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:16:53.805 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:16:53.805 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:16:53.810 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-03-12T03:16:53.814 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T03:16:53.814 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-03-12T03:16:53.814 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:16:53.814 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:16:53.814 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:16:53.814 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:16:53.814 INFO:teuthology.run_tasks:Running task pexec... 2024-03-12T03:16:53.824 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-12T03:16:53.824 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi007.front.sepia.ceph.com 2024-03-12T03:16:53.824 DEBUG:teuthology.orchestra.run.smithi007:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-12T03:16:53.825 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi033.front.sepia.ceph.com 2024-03-12T03:16:53.825 DEBUG:teuthology.orchestra.run.smithi033:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-12T03:16:53.825 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi170.front.sepia.ceph.com 2024-03-12T03:16:53.825 DEBUG:teuthology.orchestra.run.smithi170:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-12T03:16:54.432 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:48 ago on Tue 12 Mar 2024 03:16:06 AM UTC. 2024-03-12T03:16:54.432 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:58 ago on Tue 12 Mar 2024 03:13:56 AM UTC. 2024-03-12T03:16:54.437 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:03:01 ago on Tue 12 Mar 2024 03:13:53 AM UTC. 2024-03-12T03:16:54.584 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-12T03:16:54.585 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-12T03:16:54.585 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-12T03:16:54.586 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-12T03:16:54.586 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-03-12T03:16:54.586 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-12T03:16:54.586 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-12T03:16:54.586 INFO:teuthology.orchestra.run.smithi007.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-12T03:16:54.586 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:16:54.586 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-12T03:16:54.587 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-12T03:16:54.587 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-12T03:16:54.587 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:16:54.587 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-12T03:16:54.587 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-03-12T03:16:54.587 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-12T03:16:54.587 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-12T03:16:54.587 INFO:teuthology.orchestra.run.smithi170.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-12T03:16:54.587 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-12T03:16:54.587 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-12T03:16:54.587 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-12T03:16:54.588 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-12T03:16:54.588 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-12T03:16:54.591 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-12T03:16:54.592 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-03-12T03:16:54.592 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-12T03:16:54.592 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-12T03:16:54.592 INFO:teuthology.orchestra.run.smithi033.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-12T03:16:54.592 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-12T03:16:54.592 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-12T03:16:54.592 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-12T03:16:54.592 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-12T03:16:54.592 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-12T03:16:54.600 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 3.1 M 2024-03-12T03:16:54.600 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 9.8 M 2024-03-12T03:16:54.600 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-12T03:16:54.604 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 3.1 M 2024-03-12T03:16:54.604 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 9.8 M 2024-03-12T03:16:54.604 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-12T03:16:54.604 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 3.1 M 2024-03-12T03:16:54.605 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 9.8 M 2024-03-12T03:16:54.605 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-12T03:16:54.949 INFO:teuthology.orchestra.run.smithi007.stdout:runc-1.1.12-2.el9.x86_64.rpm 11 MB/s | 3.1 MB 00:00 2024-03-12T03:16:54.949 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:16:54.949 INFO:teuthology.orchestra.run.smithi007.stdout:Total 9.0 MB/s | 3.1 MB 00:00 2024-03-12T03:16:54.950 INFO:teuthology.orchestra.run.smithi033.stdout:runc-1.1.12-2.el9.x86_64.rpm 10 MB/s | 3.1 MB 00:00 2024-03-12T03:16:54.950 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:16:54.950 INFO:teuthology.orchestra.run.smithi033.stdout:Total 9.0 MB/s | 3.1 MB 00:00 2024-03-12T03:16:54.986 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-12T03:16:54.987 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-12T03:16:54.991 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-12T03:16:54.991 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-12T03:16:54.993 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-12T03:16:54.993 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-12T03:16:55.073 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-12T03:16:55.073 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-12T03:16:55.075 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-12T03:16:55.076 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-12T03:16:55.078 INFO:teuthology.orchestra.run.smithi170.stdout:runc-1.1.12-2.el9.x86_64.rpm 7.7 MB/s | 3.1 MB 00:00 2024-03-12T03:16:55.078 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:16:55.078 INFO:teuthology.orchestra.run.smithi170.stdout:Total 6.6 MB/s | 3.1 MB 00:00 2024-03-12T03:16:55.113 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-12T03:16:55.119 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-12T03:16:55.119 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-12T03:16:55.200 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-12T03:16:55.200 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-12T03:16:55.301 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-12T03:16:55.307 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-12T03:16:55.398 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T03:16:55.403 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T03:16:55.421 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-12T03:16:55.523 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T03:16:55.980 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T03:16:55.991 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T03:16:56.113 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T03:16:56.300 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T03:16:56.300 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-12T03:16:56.300 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-12T03:16:56.300 INFO:teuthology.orchestra.run.smithi033.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-12T03:16:56.300 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-12T03:16:56.301 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-12T03:16:56.366 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T03:16:56.367 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:16:56.367 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-12T03:16:56.367 INFO:teuthology.orchestra.run.smithi007.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-12T03:16:56.367 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:16:56.367 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-12T03:16:56.441 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-12T03:16:56.441 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-12T03:16:56.441 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-12T03:16:56.441 INFO:teuthology.orchestra.run.smithi170.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-12T03:16:56.441 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-12T03:16:56.441 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-12T03:16:56.556 DEBUG:teuthology.parallel:result is None 2024-03-12T03:16:56.595 DEBUG:teuthology.parallel:result is None 2024-03-12T03:16:56.690 DEBUG:teuthology.parallel:result is None 2024-03-12T03:16:56.690 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-12T03:16:56.701 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-12T03:16:56.701 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-12T03:16:56.701 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:16:56.716 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T03:16:56.716 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2024-03-12T03:16:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T03:16:56.770 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:16:56.771 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-12T03:16:56.771 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:16:56.771 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:16:56.771 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-12 03:16:52.054777407 +0000 2024-03-12T03:16:56.771 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-12 03:16:30.544105659 +0000 2024-03-12T03:16:56.771 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-12 03:16:30.544105659 +0000 2024-03-12T03:16:56.771 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-12 03:16:30.544105659 +0000 2024-03-12T03:16:56.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T03:16:56.835 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-12T03:16:56.835 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-12T03:16:56.835 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000100681 s, 5.1 MB/s 2024-03-12T03:16:56.836 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T03:16:56.892 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2024-03-12T03:16:56.946 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T03:16:56.946 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:16:56.947 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-12T03:16:56.947 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:16:56.947 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:16:56.947 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-12 03:16:52.054777407 +0000 2024-03-12T03:16:56.947 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-12 03:16:30.883100486 +0000 2024-03-12T03:16:56.947 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-12 03:16:30.883100486 +0000 2024-03-12T03:16:56.947 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-12 03:16:30.883100486 +0000 2024-03-12T03:16:56.947 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T03:16:57.011 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-12T03:16:57.011 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-12T03:16:57.011 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 8.3471e-05 s, 6.1 MB/s 2024-03-12T03:16:57.012 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T03:16:57.067 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2024-03-12T03:16:57.122 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T03:16:57.122 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:16:57.122 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-12T03:16:57.122 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:16:57.122 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:16:57.122 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-12 03:16:52.054777407 +0000 2024-03-12T03:16:57.122 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-12 03:16:31.207095542 +0000 2024-03-12T03:16:57.122 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-12 03:16:31.207095542 +0000 2024-03-12T03:16:57.123 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-12 03:16:31.207095542 +0000 2024-03-12T03:16:57.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T03:16:57.188 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-12T03:16:57.188 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-12T03:16:57.188 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.4516e-05 s, 5.4 MB/s 2024-03-12T03:16:57.189 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T03:16:57.244 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2024-03-12T03:16:57.298 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T03:16:57.299 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:16:57.299 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-12T03:16:57.299 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:16:57.299 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:16:57.299 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-12 03:16:52.054777407 +0000 2024-03-12T03:16:57.299 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-12 03:16:31.541090445 +0000 2024-03-12T03:16:57.299 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-12 03:16:31.541090445 +0000 2024-03-12T03:16:57.299 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-12 03:16:31.541090445 +0000 2024-03-12T03:16:57.299 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T03:16:57.364 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-12T03:16:57.364 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-12T03:16:57.364 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 7.9911e-05 s, 6.4 MB/s 2024-03-12T03:16:57.365 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T03:16:57.420 DEBUG:teuthology.orchestra.run.smithi007:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-12T03:16:57.567 INFO:teuthology.orchestra.run.smithi007.stdout:loop 2024-03-12T03:16:57.568 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_1... 2024-03-12T03:16:57.568 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-12T03:16:57.606 INFO:teuthology.orchestra.run.smithi007.stdout:1 2024-03-12T03:16:57.643 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_11 2024-03-12T03:16:57.672 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_2... 2024-03-12T03:16:57.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-12T03:16:57.750 INFO:teuthology.orchestra.run.smithi007.stdout:1 2024-03-12T03:16:57.787 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_21 2024-03-12T03:16:57.816 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_3... 2024-03-12T03:16:57.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-12T03:16:57.894 INFO:teuthology.orchestra.run.smithi007.stdout:1 2024-03-12T03:16:57.931 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_31 2024-03-12T03:16:57.959 INFO:tasks.nvme_loop:Connecting nvme_loop smithi007:/dev/vg_nvme/lv_4... 2024-03-12T03:16:57.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-12T03:16:58.037 INFO:teuthology.orchestra.run.smithi007.stdout:1 2024-03-12T03:16:58.074 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/vg_nvme/lv_41 2024-03-12T03:16:58.109 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-12T03:16:58.109 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:16:58.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo nvme list 2024-03-12T03:16:58.228 INFO:teuthology.orchestra.run.smithi007.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-12T03:16:58.228 INFO:teuthology.orchestra.run.smithi007.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-12T03:16:58.228 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme0n1 PHMB751300GE480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-03-12T03:16:58.228 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme1n1 cdb9c672fe801c2263e2 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T03:16:58.228 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme2n1 4b33db6b75080daa5b11 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T03:16:58.228 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme3n1 72c4ca82d801ee31dd06 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T03:16:58.228 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/nvme4n1 e0d0d19c92fa36667d17 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T03:16:58.229 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-12T03:16:58.229 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-12T03:16:58.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/scratch_devs 2024-03-12T03:16:58.296 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-12T03:16:58.296 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:16:58.314 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T03:16:58.315 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2024-03-12T03:16:58.372 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T03:16:58.372 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:16:58.372 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-12T03:16:58.372 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:16:58.372 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:16:58.372 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-12 03:16:52.059926073 +0000 2024-03-12T03:16:58.372 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-12 03:14:20.372402895 +0000 2024-03-12T03:16:58.372 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-12 03:14:20.372402895 +0000 2024-03-12T03:16:58.372 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-12 03:14:20.372402895 +0000 2024-03-12T03:16:58.373 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T03:16:58.441 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-12T03:16:58.441 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-12T03:16:58.441 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000282255 s, 1.8 MB/s 2024-03-12T03:16:58.442 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T03:16:58.499 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2024-03-12T03:16:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T03:16:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:16:58.556 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-12T03:16:58.556 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:16:58.556 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:16:58.556 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-12 03:16:52.059926073 +0000 2024-03-12T03:16:58.556 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-12 03:14:20.711397275 +0000 2024-03-12T03:16:58.556 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-12 03:14:20.711397275 +0000 2024-03-12T03:16:58.556 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-12 03:14:20.711397275 +0000 2024-03-12T03:16:58.556 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T03:16:58.623 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-12T03:16:58.623 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-12T03:16:58.624 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000214551 s, 2.4 MB/s 2024-03-12T03:16:58.625 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T03:16:58.681 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2024-03-12T03:16:58.736 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T03:16:58.737 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:16:58.737 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-12T03:16:58.737 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:16:58.737 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:16:58.737 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-12 03:16:52.059926073 +0000 2024-03-12T03:16:58.737 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-12 03:14:21.057391540 +0000 2024-03-12T03:16:58.737 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-12 03:14:21.057391540 +0000 2024-03-12T03:16:58.737 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-12 03:14:21.057391540 +0000 2024-03-12T03:16:58.737 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T03:16:58.804 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-12T03:16:58.804 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-12T03:16:58.804 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000207272 s, 2.5 MB/s 2024-03-12T03:16:58.805 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T03:16:58.861 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2024-03-12T03:16:58.916 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T03:16:58.916 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:16:58.916 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-12T03:16:58.916 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:16:58.916 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:16:58.917 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-12 03:16:52.059926073 +0000 2024-03-12T03:16:58.917 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-12 03:14:21.400385854 +0000 2024-03-12T03:16:58.917 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-12 03:14:21.400385854 +0000 2024-03-12T03:16:58.917 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-12 03:14:21.400385854 +0000 2024-03-12T03:16:58.917 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T03:16:58.982 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-12T03:16:58.982 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-12T03:16:58.982 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000163479 s, 3.1 MB/s 2024-03-12T03:16:58.983 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T03:16:59.039 DEBUG:teuthology.orchestra.run.smithi033:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-12T03:16:59.175 INFO:teuthology.orchestra.run.smithi033.stdout:loop 2024-03-12T03:16:59.176 INFO:tasks.nvme_loop:Connecting nvme_loop smithi033:/dev/vg_nvme/lv_1... 2024-03-12T03:16:59.177 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-12T03:16:59.214 INFO:teuthology.orchestra.run.smithi033.stdout:1 2024-03-12T03:16:59.251 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/vg_nvme/lv_11 2024-03-12T03:16:59.280 INFO:tasks.nvme_loop:Connecting nvme_loop smithi033:/dev/vg_nvme/lv_2... 2024-03-12T03:16:59.281 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-12T03:16:59.359 INFO:teuthology.orchestra.run.smithi033.stdout:1 2024-03-12T03:16:59.410 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/vg_nvme/lv_21 2024-03-12T03:16:59.439 INFO:tasks.nvme_loop:Connecting nvme_loop smithi033:/dev/vg_nvme/lv_3... 2024-03-12T03:16:59.439 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-12T03:16:59.517 INFO:teuthology.orchestra.run.smithi033.stdout:1 2024-03-12T03:16:59.556 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/vg_nvme/lv_31 2024-03-12T03:16:59.585 INFO:tasks.nvme_loop:Connecting nvme_loop smithi033:/dev/vg_nvme/lv_4... 2024-03-12T03:16:59.585 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-12T03:16:59.664 INFO:teuthology.orchestra.run.smithi033.stdout:1 2024-03-12T03:16:59.703 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/vg_nvme/lv_41 2024-03-12T03:16:59.733 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-12T03:16:59.733 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:16:59.788 DEBUG:teuthology.orchestra.run.smithi033:> sudo nvme list 2024-03-12T03:16:59.853 INFO:teuthology.orchestra.run.smithi033.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-12T03:16:59.854 INFO:teuthology.orchestra.run.smithi033.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-12T03:16:59.854 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/nvme0n1 CVFT5330008Q400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-12T03:16:59.854 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/nvme1n1 386594f789e6b989396c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T03:16:59.854 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/nvme2n1 acaba4db481862bf6cc5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T03:16:59.854 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/nvme3n1 c1863f6c2613c5dc9a48 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T03:16:59.854 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/nvme4n1 a587843b9b7207625e5c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T03:16:59.855 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-12T03:16:59.855 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-12T03:16:59.855 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/scratch_devs 2024-03-12T03:16:59.922 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-12T03:16:59.922 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:16:59.940 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T03:16:59.940 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2024-03-12T03:16:59.998 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T03:16:59.998 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:16:59.998 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-12T03:16:59.998 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:16:59.998 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:16:59.998 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-12 03:16:52.069011892 +0000 2024-03-12T03:16:59.998 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-12 03:14:18.091769283 +0000 2024-03-12T03:16:59.998 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-12 03:14:18.091769283 +0000 2024-03-12T03:16:59.998 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-12 03:14:18.091769283 +0000 2024-03-12T03:16:59.998 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T03:17:00.067 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-12T03:17:00.099 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-12T03:17:00.099 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 8.9905e-05 s, 5.7 MB/s 2024-03-12T03:17:00.100 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T03:17:00.157 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2024-03-12T03:17:00.214 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T03:17:00.215 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:17:00.215 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-12T03:17:00.215 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:17:00.215 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:17:00.215 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-12 03:16:52.069011892 +0000 2024-03-12T03:17:00.215 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-12 03:14:18.447760583 +0000 2024-03-12T03:17:00.215 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-12 03:14:18.447760583 +0000 2024-03-12T03:17:00.215 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-12 03:14:18.447760583 +0000 2024-03-12T03:17:00.216 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T03:17:00.282 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-12T03:17:00.282 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-12T03:17:00.283 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000113487 s, 4.5 MB/s 2024-03-12T03:17:00.284 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T03:17:00.341 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2024-03-12T03:17:00.397 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T03:17:00.397 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:17:00.397 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-12T03:17:00.397 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:17:00.397 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:17:00.397 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-12 03:16:52.069011892 +0000 2024-03-12T03:17:00.397 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-12 03:14:18.769752714 +0000 2024-03-12T03:17:00.397 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-12 03:14:18.769752714 +0000 2024-03-12T03:17:00.397 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-12 03:14:18.769752714 +0000 2024-03-12T03:17:00.397 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T03:17:00.463 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-12T03:17:00.463 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-12T03:17:00.463 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000120363 s, 4.3 MB/s 2024-03-12T03:17:00.464 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T03:17:00.521 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2024-03-12T03:17:00.576 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T03:17:00.576 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:17:00.576 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-12T03:17:00.576 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:17:00.576 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:17:00.576 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-12 03:16:52.069011892 +0000 2024-03-12T03:17:00.576 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-12 03:14:19.099744649 +0000 2024-03-12T03:17:00.576 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-12 03:14:19.099744649 +0000 2024-03-12T03:17:00.576 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-12 03:14:19.099744649 +0000 2024-03-12T03:17:00.576 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T03:17:00.642 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-12T03:17:00.642 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-12T03:17:00.642 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 8.0067e-05 s, 6.4 MB/s 2024-03-12T03:17:00.643 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T03:17:00.699 DEBUG:teuthology.orchestra.run.smithi170:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-12T03:17:00.850 INFO:teuthology.orchestra.run.smithi170.stdout:loop 2024-03-12T03:17:00.851 INFO:tasks.nvme_loop:Connecting nvme_loop smithi170:/dev/vg_nvme/lv_1... 2024-03-12T03:17:00.851 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-12T03:17:00.889 INFO:teuthology.orchestra.run.smithi170.stdout:1 2024-03-12T03:17:00.926 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/vg_nvme/lv_11 2024-03-12T03:17:00.955 INFO:tasks.nvme_loop:Connecting nvme_loop smithi170:/dev/vg_nvme/lv_2... 2024-03-12T03:17:00.955 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-12T03:17:01.035 INFO:teuthology.orchestra.run.smithi170.stdout:1 2024-03-12T03:17:01.074 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/vg_nvme/lv_21 2024-03-12T03:17:01.103 INFO:tasks.nvme_loop:Connecting nvme_loop smithi170:/dev/vg_nvme/lv_3... 2024-03-12T03:17:01.103 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-12T03:17:01.182 INFO:teuthology.orchestra.run.smithi170.stdout:1 2024-03-12T03:17:01.221 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/vg_nvme/lv_31 2024-03-12T03:17:01.250 INFO:tasks.nvme_loop:Connecting nvme_loop smithi170:/dev/vg_nvme/lv_4... 2024-03-12T03:17:01.250 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-12T03:17:01.331 INFO:teuthology.orchestra.run.smithi170.stdout:1 2024-03-12T03:17:01.370 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/vg_nvme/lv_41 2024-03-12T03:17:01.407 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-12T03:17:01.407 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:17:01.461 DEBUG:teuthology.orchestra.run.smithi170:> sudo nvme list 2024-03-12T03:17:01.527 INFO:teuthology.orchestra.run.smithi170.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-12T03:17:01.527 INFO:teuthology.orchestra.run.smithi170.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-12T03:17:01.528 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme0n1 PHMB7476002R480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-03-12T03:17:01.528 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme1n1 345ab9225a67e14d3da0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T03:17:01.528 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme2n1 5351117ee63a7d3743c0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T03:17:01.528 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme3n1 1824f99b508c04c684b9 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T03:17:01.528 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme4n1 c9338f697901a0741dd1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-12T03:17:01.529 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-12T03:17:01.529 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-12T03:17:01.529 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/scratch_devs 2024-03-12T03:17:01.596 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-12T03:17:01.690 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'global': {'mon election default strategy': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'} 2024-03-12T03:17:01.690 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:17:01.691 INFO:tasks.cephadm:Cluster fsid is fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:17:01.691 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-12T03:17:01.691 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.7', 'mon.b': '172.21.15.33', 'mon.c': '172.21.15.170'} 2024-03-12T03:17:01.691 INFO:tasks.cephadm:First mon is mon.a on smithi007 2024-03-12T03:17:01.692 INFO:tasks.cephadm:First mgr is y 2024-03-12T03:17:01.692 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-12T03:17:01.692 DEBUG:teuthology.orchestra.run.smithi007:> sudo hostname $(hostname -s) 2024-03-12T03:17:01.726 DEBUG:teuthology.orchestra.run.smithi033:> sudo hostname $(hostname -s) 2024-03-12T03:17:01.759 DEBUG:teuthology.orchestra.run.smithi170:> sudo hostname $(hostname -s) 2024-03-12T03:17:01.786 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-12T03:17:01.787 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:17:01.933 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54565/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1979-g2b0101eb', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1979.g2b0101eb'}, 'url': 'https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-11 00:47:00.272739', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-12T03:17:02.087 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref main, sha1 2b0101eb74d60610d544cb8a473e23a35ac1cc0b from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:17:02.088 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm 2024-03-12T03:17:02.088 DEBUG:teuthology.orchestra.run.smithi007:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T03:17:03.075 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r--. 1 ubuntu ubuntu 767461 Mar 12 03:17 /home/ubuntu/cephtest/cephadm 2024-03-12T03:17:03.075 DEBUG:teuthology.orchestra.run.smithi033:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T03:17:04.089 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-r--r--. 1 ubuntu ubuntu 767461 Mar 12 03:17 /home/ubuntu/cephtest/cephadm 2024-03-12T03:17:04.090 DEBUG:teuthology.orchestra.run.smithi170:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T03:17:04.519 INFO:teuthology.orchestra.run.smithi170.stdout:-rw-r--r--. 1 ubuntu ubuntu 767461 Mar 12 03:17 /home/ubuntu/cephtest/cephadm 2024-03-12T03:17:04.520 DEBUG:teuthology.orchestra.run.smithi007:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T03:17:04.538 DEBUG:teuthology.orchestra.run.smithi033:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T03:17:04.556 DEBUG:teuthology.orchestra.run.smithi170:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T03:17:04.587 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b on all hosts... 2024-03-12T03:17:04.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-12T03:17:04.590 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-12T03:17:04.599 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-12T03:17:05.435 INFO:teuthology.orchestra.run.smithi007.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T03:17:05.471 INFO:teuthology.orchestra.run.smithi033.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T03:17:05.472 INFO:teuthology.orchestra.run.smithi170.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T03:17:32.729 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-12T03:17:32.729 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-12T03:17:32.729 INFO:teuthology.orchestra.run.smithi033.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-12T03:17:32.729 INFO:teuthology.orchestra.run.smithi033.stdout: "repo_digests": [ 2024-03-12T03:17:32.729 INFO:teuthology.orchestra.run.smithi033.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-12T03:17:32.729 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-03-12T03:17:32.729 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-12T03:18:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-12T03:18:03.175 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-12T03:18:03.175 INFO:teuthology.orchestra.run.smithi007.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-12T03:18:03.175 INFO:teuthology.orchestra.run.smithi007.stdout: "repo_digests": [ 2024-03-12T03:18:03.175 INFO:teuthology.orchestra.run.smithi007.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-12T03:18:03.175 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-03-12T03:18:03.175 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-12T03:18:08.303 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-03-12T03:18:08.304 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-12T03:18:08.304 INFO:teuthology.orchestra.run.smithi170.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-12T03:18:08.304 INFO:teuthology.orchestra.run.smithi170.stdout: "repo_digests": [ 2024-03-12T03:18:08.304 INFO:teuthology.orchestra.run.smithi170.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-12T03:18:08.304 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-03-12T03:18:08.304 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-03-12T03:18:08.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph 2024-03-12T03:18:08.367 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph 2024-03-12T03:18:08.403 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph 2024-03-12T03:18:08.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /etc/ceph 2024-03-12T03:18:08.465 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /etc/ceph 2024-03-12T03:18:08.496 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /etc/ceph 2024-03-12T03:18:08.525 INFO:tasks.cephadm:Writing seed config... 2024-03-12T03:18:08.526 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-12T03:18:08.526 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-12T03:18:08.526 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-03-12T03:18:08.526 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-12T03:18:08.526 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-12T03:18:08.526 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-12T03:18:08.527 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-12T03:18:08.527 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-12T03:18:08.527 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-12T03:18:08.527 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-12T03:18:08.527 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-12T03:18:08.527 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-12T03:18:08.543 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = fe67b1e4-e01e-11ee-95c8-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-12T03:18:08.544 DEBUG:teuthology.orchestra.run.smithi007:mon.a> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mon.a.service 2024-03-12T03:18:08.586 DEBUG:teuthology.orchestra.run.smithi007:mgr.y> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mgr.y.service 2024-03-12T03:18:08.629 INFO:tasks.cephadm:Bootstrapping... 2024-03-12T03:18:08.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b -v bootstrap --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.7 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:18:08.827 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:18:08.827 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b', '-v', 'bootstrap', '--fsid', 'fe67b1e4-e01e-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.7', '--skip-admin-label'] 2024-03-12T03:18:08.852 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T03:18:08.853 INFO:teuthology.orchestra.run.smithi007.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-12T03:18:08.853 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying podman|docker is present... 2024-03-12T03:18:08.877 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T03:18:08.877 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying lvm2 is present... 2024-03-12T03:18:08.877 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying time synchronization is in place... 2024-03-12T03:18:08.884 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-12T03:18:08.884 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T03:18:08.889 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-12T03:18:08.890 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-03-12T03:18:08.896 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout enabled 2024-03-12T03:18:08.902 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout active 2024-03-12T03:18:08.902 INFO:teuthology.orchestra.run.smithi007.stdout:Unit chronyd.service is enabled and running 2024-03-12T03:18:08.902 INFO:teuthology.orchestra.run.smithi007.stdout:Repeating the final host check... 2024-03-12T03:18:08.925 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T03:18:08.925 INFO:teuthology.orchestra.run.smithi007.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-12T03:18:08.925 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl is present 2024-03-12T03:18:08.925 INFO:teuthology.orchestra.run.smithi007.stdout:lvcreate is present 2024-03-12T03:18:08.931 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-12T03:18:08.931 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T03:18:08.937 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-12T03:18:08.937 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-03-12T03:18:08.943 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout enabled 2024-03-12T03:18:08.949 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout active 2024-03-12T03:18:08.949 INFO:teuthology.orchestra.run.smithi007.stdout:Unit chronyd.service is enabled and running 2024-03-12T03:18:08.949 INFO:teuthology.orchestra.run.smithi007.stdout:Host looks OK 2024-03-12T03:18:08.950 INFO:teuthology.orchestra.run.smithi007.stdout:Cluster fsid: fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:18:08.950 INFO:teuthology.orchestra.run.smithi007.stdout:Acquiring lock 140465700910464 on /run/cephadm/fe67b1e4-e01e-11ee-95c8-87774f69a715.lock 2024-03-12T03:18:08.950 INFO:teuthology.orchestra.run.smithi007.stdout:Lock 140465700910464 acquired on /run/cephadm/fe67b1e4-e01e-11ee-95c8-87774f69a715.lock 2024-03-12T03:18:08.950 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying IP 172.21.15.7 port 3300 ... 2024-03-12T03:18:08.950 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying IP 172.21.15.7 port 6789 ... 2024-03-12T03:18:08.950 INFO:teuthology.orchestra.run.smithi007.stdout:Base mon IP(s) is [172.21.15.7:3300, 172.21.15.7:6789], mon addrv is [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2024-03-12T03:18:08.953 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.7 metric 100 2024-03-12T03:18:08.953 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.7 metric 100 2024-03-12T03:18:08.956 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-12T03:18:08.956 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-12T03:18:08.956 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-12T03:18:08.958 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-12T03:18:08.958 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-12T03:18:08.959 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-12T03:18:08.959 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-12T03:18:08.959 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15ea/64 scope link noprefixroute 2024-03-12T03:18:08.959 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-12T03:18:08.959 INFO:teuthology.orchestra.run.smithi007.stdout:Mon IP `172.21.15.7` is in CIDR network `172.21.0.0/20` 2024-03-12T03:18:08.959 INFO:teuthology.orchestra.run.smithi007.stdout:Mon IP `172.21.15.7` is in CIDR network `172.21.0.0/20` 2024-03-12T03:18:08.959 INFO:teuthology.orchestra.run.smithi007.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-12T03:18:08.960 INFO:teuthology.orchestra.run.smithi007.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-12T03:18:08.960 INFO:teuthology.orchestra.run.smithi007.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T03:18:09.443 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-12T03:18:09.443 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-12T03:18:09.443 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Getting image source signatures 2024-03-12T03:18:09.443 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying blob sha256:4392e0adce526b0883d65bbedc13478c05c0e708702f1c7015b93fe77c162cd5 2024-03-12T03:18:09.443 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-12T03:18:09.443 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying config sha256:4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-12T03:18:09.443 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-12T03:18:10.448 INFO:teuthology.orchestra.run.smithi007.stdout:ceph: stdout ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-12T03:18:10.448 INFO:teuthology.orchestra.run.smithi007.stdout:Ceph version: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-12T03:18:10.448 INFO:teuthology.orchestra.run.smithi007.stdout:Extracting ceph user uid/gid from container image... 2024-03-12T03:18:11.320 INFO:teuthology.orchestra.run.smithi007.stdout:stat: stdout 167 167 2024-03-12T03:18:11.320 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial keys... 2024-03-12T03:18:12.300 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQBzye9lbjz2KRAARlmvqIlWua6AAykfLpq3FA== 2024-03-12T03:18:13.356 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQB0ye9lodnQJxAAyUNY/qj4t2B43rWBaiR1OQ== 2024-03-12T03:18:14.312 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQB1ye9lKrszKxAAy0Tq0VGTUI/vqVBBMa6xmQ== 2024-03-12T03:18:14.313 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial monmap... 2024-03-12T03:18:15.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T03:18:15.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-12T03:18:15.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:18:15.261 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T03:18:15.261 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool for a [v2:172.21.15.7:3300,v1:172.21.15.7:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T03:18:15.261 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = quincy 2024-03-12T03:18:15.261 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: set fsid to fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:18:15.261 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T03:18:15.261 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:18:15.261 INFO:teuthology.orchestra.run.smithi007.stdout:Creating mon... 2024-03-12T03:18:16.224 INFO:teuthology.orchestra.run.smithi007.stdout:create mon.a on 2024-03-12T03:18:16.573 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-12T03:18:16.754 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715.target. 2024-03-12T03:18:16.754 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715.target. 2024-03-12T03:18:16.985 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mon.a 2024-03-12T03:18:16.985 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to reset failed state of unit ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-12T03:18:17.145 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715.target.wants/ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@.service. 2024-03-12T03:18:17.705 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:18:17.705 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-03-12T03:18:17.712 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:18:17.712 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-03-12T03:18:17.712 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-03-12T03:18:17.712 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to enable service . firewalld.service is not available 2024-03-12T03:18:17.712 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mon to start... 2024-03-12T03:18:17.712 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mon... 2024-03-12T03:18:18.752 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout cluster: 2024-03-12T03:18:18.752 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout id: fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:18:18.752 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-12T03:18:18.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-12T03:18:18.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout services: 2024-03-12T03:18:18.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.572632s) 2024-03-12T03:18:18.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-12T03:18:18.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-12T03:18:18.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-12T03:18:18.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout data: 2024-03-12T03:18:18.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-12T03:18:18.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-12T03:18:18.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-12T03:18:18.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout pgs: 2024-03-12T03:18:18.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-12T03:18:18.753 INFO:teuthology.orchestra.run.smithi007.stdout:mon is available 2024-03-12T03:18:18.753 INFO:teuthology.orchestra.run.smithi007.stdout:Assimilating anything we can from ceph.conf... 2024-03-12T03:18:19.838 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:19 smithi007 ceph-mon[25312]: from='client.? 172.21.15.7:0/3031138831' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-12T03:18:19.842 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-12T03:18:19.842 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [global] 2024-03-12T03:18:19.843 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout fsid = fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:18:19.843 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2024-03-12T03:18:19.843 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-12T03:18:19.843 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-12T03:18:19.843 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-12T03:18:19.843 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-12T03:18:19.843 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-12T03:18:19.843 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-12T03:18:19.843 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-03-12T03:18:19.843 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-12T03:18:19.843 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-12T03:18:19.843 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [osd] 2024-03-12T03:18:19.843 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-12T03:18:19.843 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-12T03:18:19.843 INFO:teuthology.orchestra.run.smithi007.stdout:Generating new minimal ceph.conf... 2024-03-12T03:18:20.914 INFO:teuthology.orchestra.run.smithi007.stdout:Restarting the monitor... 2024-03-12T03:18:20.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:20 smithi007 ceph-mon[25312]: from='client.? 172.21.15.7:0/1482713989' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:21.184 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:20 smithi007 systemd[1]: Stopping Ceph mon.a for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:18:21.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:21 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-a[25289]: 2024-03-12T03:18:21.181+0000 7f899f168700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:18:21.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:21 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-a[25289]: 2024-03-12T03:18:21.181+0000 7f899f168700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-12T03:18:21.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:21 smithi007 podman[25645]: 2024-03-12 03:18:21.386840028 +0000 UTC m=+0.311977668 container died 6ee47e06d1ad31014d69a82ba0104d125fa15367e45ba3ce4a95dcf8e0864ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-12T03:18:22.014 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:21 smithi007 podman[25645]: 2024-03-12 03:18:21.737219164 +0000 UTC m=+0.662356803 container cleanup 6ee47e06d1ad31014d69a82ba0104d125fa15367e45ba3ce4a95dcf8e0864ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, io.buildah.version=1.33.5) 2024-03-12T03:18:22.014 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:21 smithi007 bash[25645]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-a 2024-03-12T03:18:22.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 podman[25664]: 2024-03-12 03:18:22.012979081 +0000 UTC m=+0.623928225 container remove 6ee47e06d1ad31014d69a82ba0104d125fa15367e45ba3ce4a95dcf8e0864ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:18:22.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-12T03:18:22.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 systemd[1]: Stopped Ceph mon.a for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:18:22.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 systemd[1]: Starting Ceph mon.a for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:18:22.629 INFO:teuthology.orchestra.run.smithi007.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-12T03:18:22.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 podman[25763]: 2024-03-12 03:18:22.281358043 +0000 UTC m=+0.017384119 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:18:22.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 podman[25763]: 2024-03-12 03:18:22.380361187 +0000 UTC m=+0.116387247 container create d8b3346c080857fc0e090db859a5f10822fc26c6ac37c7ce059fb8fd209ce621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:18:22.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 podman[25763]: 2024-03-12 03:18:22.558149721 +0000 UTC m=+0.294175782 container init d8b3346c080857fc0e090db859a5f10822fc26c6ac37c7ce059fb8fd209ce621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-12T03:18:22.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 podman[25763]: 2024-03-12 03:18:22.563284142 +0000 UTC m=+0.299310203 container start d8b3346c080857fc0e090db859a5f10822fc26c6ac37c7ce059fb8fd209ce621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:18:22.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T03:18:22.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 6 2024-03-12T03:18:22.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: pidfile_write: ignore empty --pid-file 2024-03-12T03:18:22.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: load: jerasure load: lrc 2024-03-12T03:18:22.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: RocksDB version: 7.9.2 2024-03-12T03:18:22.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Git sha 0 2024-03-12T03:18:22.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-12T03:18:22.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: DB SUMMARY 2024-03-12T03:18:22.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: DB Session ID: 9XZI2SVT5B0COMGNFI8G 2024-03-12T03:18:22.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: CURRENT file: CURRENT 2024-03-12T03:18:22.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T03:18:22.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-12T03:18:22.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-12T03:18:22.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84601 ; 2024-03-12T03:18:22.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.error_if_exists: 0 2024-03-12T03:18:22.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.create_if_missing: 0 2024-03-12T03:18:22.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.paranoid_checks: 1 2024-03-12T03:18:22.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T03:18:22.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T03:18:22.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T03:18:22.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.env: 0x55c8eb757b20 2024-03-12T03:18:22.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T03:18:22.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.info_log: 0x55c8ec56b8e0 2024-03-12T03:18:22.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T03:18:22.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.statistics: (nil) 2024-03-12T03:18:22.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.use_fsync: 0 2024-03-12T03:18:22.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_log_file_size: 0 2024-03-12T03:18:22.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T03:18:22.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T03:18:22.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T03:18:22.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T03:18:22.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.allow_fallocate: 1 2024-03-12T03:18:22.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T03:18:22.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T03:18:22.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.use_direct_reads: 0 2024-03-12T03:18:22.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T03:18:22.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T03:18:22.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.db_log_dir: 2024-03-12T03:18:22.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.wal_dir: 2024-03-12T03:18:22.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T03:18:22.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T03:18:22.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T03:18:22.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T03:18:22.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T03:18:22.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T03:18:22.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T03:18:22.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T03:18:22.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.write_buffer_manager: 0x55c8ec53f400 2024-03-12T03:18:22.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T03:18:22.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T03:18:22.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T03:18:22.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T03:18:22.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T03:18:22.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T03:18:22.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T03:18:22.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T03:18:22.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.unordered_write: 0 2024-03-12T03:18:22.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T03:18:22.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T03:18:22.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T03:18:22.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T03:18:22.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.row_cache: None 2024-03-12T03:18:22.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.wal_filter: None 2024-03-12T03:18:22.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T03:18:22.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T03:18:22.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.two_write_queues: 0 2024-03-12T03:18:22.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T03:18:22.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.wal_compression: 0 2024-03-12T03:18:22.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.atomic_flush: 0 2024-03-12T03:18:22.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T03:18:22.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T03:18:22.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T03:18:22.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.log_readahead_size: 0 2024-03-12T03:18:22.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T03:18:22.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T03:18:22.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T03:18:22.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T03:18:22.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T03:18:22.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T03:18:22.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T03:18:22.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_background_jobs: 2 2024-03-12T03:18:22.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_background_compactions: -1 2024-03-12T03:18:22.671 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_subcompactions: 1 2024-03-12T03:18:22.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T03:18:22.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T03:18:22.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T03:18:22.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T03:18:22.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T03:18:22.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T03:18:22.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T03:18:22.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T03:18:22.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_open_files: -1 2024-03-12T03:18:22.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T03:18:22.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T03:18:22.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T03:18:22.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T03:18:22.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_background_flushes: -1 2024-03-12T03:18:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Compression algorithms supported: 2024-03-12T03:18:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T03:18:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: kZSTD supported: 0 2024-03-12T03:18:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: kXpressCompression supported: 0 2024-03-12T03:18:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T03:18:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: kZlibCompression supported: 1 2024-03-12T03:18:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: kSnappyCompression supported: 1 2024-03-12T03:18:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: kLZ4Compression supported: 1 2024-03-12T03:18:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: kBZip2Compression supported: 0 2024-03-12T03:18:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T03:18:22.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.merge_operator: 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compaction_filter: None 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c8ec56b9e0) 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks: 1 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_top_level_index_and_filter: 1 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_type: 0 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_index_type: 0 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_shortening: 1 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: checksum: 4 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: no_block_cache: 0 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache: 0x55c8ec571090 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_name: BinnedLRUCache 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_options: 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: capacity : 536870912 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_shard_bits : 4 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: strict_capacity_limit : 0 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: high_pri_pool_ratio: 0.000 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_compressed: (nil) 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: persistent_cache: (nil) 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size: 4096 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size_deviation: 10 2024-03-12T03:18:22.675 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_restart_interval: 16 2024-03-12T03:18:22.676 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_block_restart_interval: 1 2024-03-12T03:18:22.676 INFO:journalctl@ceph.mon.a.smithi007.stdout: metadata_block_size: 4096 2024-03-12T03:18:22.676 INFO:journalctl@ceph.mon.a.smithi007.stdout: partition_filters: 0 2024-03-12T03:18:22.676 INFO:journalctl@ceph.mon.a.smithi007.stdout: use_delta_encoding: 1 2024-03-12T03:18:22.676 INFO:journalctl@ceph.mon.a.smithi007.stdout: filter_policy: bloomfilter 2024-03-12T03:18:22.676 INFO:journalctl@ceph.mon.a.smithi007.stdout: whole_key_filtering: 1 2024-03-12T03:18:22.676 INFO:journalctl@ceph.mon.a.smithi007.stdout: verify_compression: 0 2024-03-12T03:18:22.676 INFO:journalctl@ceph.mon.a.smithi007.stdout: read_amp_bytes_per_bit: 0 2024-03-12T03:18:22.676 INFO:journalctl@ceph.mon.a.smithi007.stdout: format_version: 5 2024-03-12T03:18:22.676 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_index_compression: 1 2024-03-12T03:18:22.676 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_align: 0 2024-03-12T03:18:22.676 INFO:journalctl@ceph.mon.a.smithi007.stdout: max_auto_readahead_size: 262144 2024-03-12T03:18:22.676 INFO:journalctl@ceph.mon.a.smithi007.stdout: prepopulate_block_cache: 0 2024-03-12T03:18:22.676 INFO:journalctl@ceph.mon.a.smithi007.stdout: initial_auto_readahead_size: 8192 2024-03-12T03:18:22.676 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compression: NoCompression 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.num_levels: 7 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:18:22.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T03:18:22.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T03:18:22.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:18:22.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:18:22.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T03:18:22.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T03:18:22.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T03:18:22.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T03:18:22.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:18:22.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:18:22.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T03:18:22.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T03:18:22.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:18:22.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T03:18:22.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T03:18:22.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.table_properties_collectors: 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.inplace_update_support: 0 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T03:18:22.680 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.bloom_locality: 0 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.max_successive_merges: 0 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.ttl: 2592000 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.enable_blob_files: false 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.min_blob_size: 0 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T03:18:22.681 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 344f19b2-462d-478b-b232-ec388d54849f 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710213502592355, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710213502593191, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81226, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79413, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73722, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710213502, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "344f19b2-462d-478b-b232-ec388d54849f", "db_session_id": "9XZI2SVT5B0COMGNFI8G", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710213502593274, "job": 1, "event": "recovery_finished"} 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c8ec664000 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: DB pointer 0x55c8ec64e000 2024-03-12T03:18:22.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T03:18:22.683 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T03:18:22.683 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** DB Stats ** 2024-03-12T03:18:22.683 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:18:22.683 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T03:18:22.683 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:18:22.683 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:18:22.683 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T03:18:22.683 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:18:22.683 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:18:22.683 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-03-12T03:18:22.683 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-03-12T03:18:22.683 INFO:journalctl@ceph.mon.a.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:18:22.683 INFO:journalctl@ceph.mon.a.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T03:18:22.683 INFO:journalctl@ceph.mon.a.smithi007.stdout: L0 2/0 81.16 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:18:22.683 INFO:journalctl@ceph.mon.a.smithi007.stdout: Sum 2/0 81.16 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative compaction: 0.00 GB write, 27.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:18:22.684 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval compaction: 0.00 GB write, 27.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:18:22.685 INFO:journalctl@ceph.mon.a.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:18:22.685 INFO:journalctl@ceph.mon.a.smithi007.stdout: Block cache BinnedLRUCache@0x55c8ec571090#6 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-12T03:18:22.685 INFO:journalctl@ceph.mon.a.smithi007.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-12T03:18:22.685 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-03-12T03:18:22.685 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:18:22.685 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: starting mon.a rank 0 at public addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] at bind addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:18:22.685 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: mon.a@-1(???) e1 preinit fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:18:22.685 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: mon.a@-1(???).mds e1 new map 2024-03-12T03:18:22.685 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: mon.a@-1(???).mds e1 print_map 2024-03-12T03:18:22.685 INFO:journalctl@ceph.mon.a.smithi007.stdout: e1 2024-03-12T03:18:22.685 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:18:22.685 INFO:journalctl@ceph.mon.a.smithi007.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T03:18:22.685 INFO:journalctl@ceph.mon.a.smithi007.stdout: legacy client fscid: -1 2024-03-12T03:18:22.685 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-03-12T03:18:22.685 INFO:journalctl@ceph.mon.a.smithi007.stdout: No filesystems configured 2024-03-12T03:18:22.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T03:18:22.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:18:22.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:18:22.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:18:22.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-12T03:18:22.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 bash[25763]: d8b3346c080857fc0e090db859a5f10822fc26c6ac37c7ce059fb8fd209ce621 2024-03-12T03:18:22.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 systemd[1]: Started Ceph mon.a for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:18:22.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-12T03:18:22.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: monmap e1: 1 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:18:22.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: fsmap 2024-03-12T03:18:22.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: osdmap e1: 0 total, 0 up, 0 in 2024-03-12T03:18:22.686 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:22 smithi007 ceph-mon[25800]: mgrmap e1: no daemons active 2024-03-12T03:18:23.836 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-12T03:18:23.857 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:18:23.858 INFO:teuthology.orchestra.run.smithi007.stdout:Creating mgr... 2024-03-12T03:18:23.858 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-12T03:18:23.859 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-12T03:18:24.114 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mgr.y 2024-03-12T03:18:24.114 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to reset failed state of unit ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-12T03:18:24.279 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715.target.wants/ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@.service. 2024-03-12T03:18:24.502 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:24 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/1381329093' entity='client.admin' 2024-03-12T03:18:24.502 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:24 smithi007 systemd[1]: Starting Ceph mgr.y for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:18:24.821 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:24 smithi007 podman[26044]: 2024-03-12 03:18:24.500323731 +0000 UTC m=+0.104035290 container create 42b9d300140d2fe41159f3b9ea09eaf15cd7f57d7111e46defb8fab00af5d075 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-12T03:18:24.821 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:24 smithi007 podman[26044]: 2024-03-12 03:18:24.414522756 +0000 UTC m=+0.018234338 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:18:24.821 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:24 smithi007 podman[26044]: 2024-03-12 03:18:24.702710416 +0000 UTC m=+0.306421982 container init 42b9d300140d2fe41159f3b9ea09eaf15cd7f57d7111e46defb8fab00af5d075 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, ceph=True, GIT_BRANCH=HEAD) 2024-03-12T03:18:24.821 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:24 smithi007 podman[26044]: 2024-03-12 03:18:24.708049662 +0000 UTC m=+0.311761221 container start 42b9d300140d2fe41159f3b9ea09eaf15cd7f57d7111e46defb8fab00af5d075 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=main-2b0101e, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-12T03:18:24.821 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:24 smithi007 bash[26044]: 42b9d300140d2fe41159f3b9ea09eaf15cd7f57d7111e46defb8fab00af5d075 2024-03-12T03:18:24.834 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:18:24.834 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-03-12T03:18:24.841 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:18:24.841 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-03-12T03:18:24.841 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-03-12T03:18:24.841 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to enable service . firewalld.service is not available 2024-03-12T03:18:24.849 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:18:24.849 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-03-12T03:18:24.856 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:18:24.856 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-03-12T03:18:24.856 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-03-12T03:18:24.856 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-12T03:18:24.856 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr to start... 2024-03-12T03:18:24.856 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr... 2024-03-12T03:18:25.098 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:24 smithi007 systemd[1]: Started Ceph mgr.y for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:18:25.373 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:25 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:25.272+0000 7fbf18356200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:18:25.373 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:25 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:25.370+0000 7fbf18356200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:18:25.744 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:25 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/3614160733' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:18:26.024 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-12T03:18:26.024 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:18:26.024 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "fe67b1e4-e01e-11ee-95c8-87774f69a715", 2024-03-12T03:18:26.024 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T03:18:26.024 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T03:18:26.024 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T03:18:26.024 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T03:18:26.024 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:26.024 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T03:18:26.024 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T03:18:26.024 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-03-12T03:18:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:18:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T03:18:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T03:18:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:18:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-12T03:18:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T03:18:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:18:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T03:18:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T03:18:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T03:18:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:18:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T03:18:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T03:18:26.025 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T03:18:26.029 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T03:18:26.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T03:18:26.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T03:18:26.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:26.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T03:18:26.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T03:18:26.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T03:18:26.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T03:18:26.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T03:18:26.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T03:18:26.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T03:18:26.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T03:18:26.030 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T03:18:26.031 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:26.031 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T03:18:26.031 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:18:26.031 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T03:18:26.031 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T03:18:26.031 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:26.031 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T03:18:26.031 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T03:18:26.031 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T03:18:26.031 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T03:18:26.031 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T03:18:26.031 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T03:18:26.032 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T03:18:26.032 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:18:26.032 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:18:26.032 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:26.032 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T03:18:26.032 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:18:26.032 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T03:18:17.656503+0000", 2024-03-12T03:18:26.032 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:18:26.032 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:26.032 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T03:18:26.032 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:18:26.032 INFO:teuthology.orchestra.run.smithi007.stdout:mgr not available, waiting (1/15)... 2024-03-12T03:18:26.288 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:26 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:26.048+0000 7fbf18356200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:18:26.288 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:26 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:26.143+0000 7fbf18356200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:18:26.598 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:26 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:26.286+0000 7fbf18356200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:18:26.598 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:26 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:26.383+0000 7fbf18356200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:18:26.598 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:26 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:26.468+0000 7fbf18356200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:18:27.032 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:26 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:26.734+0000 7fbf18356200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:18:27.033 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:26 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:26.846+0000 7fbf18356200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:18:27.033 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:26 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:26.934+0000 7fbf18356200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:18:27.033 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:27 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:27.030+0000 7fbf18356200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:18:27.348 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:27 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:27.207+0000 7fbf18356200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:18:28.598 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:28 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:28.223+0000 7fbf18356200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:18:28.598 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:28 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:28.358+0000 7fbf18356200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:18:28.987 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:28 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:28.694+0000 7fbf18356200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:18:28.988 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:28 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:28.786+0000 7fbf18356200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:18:28.988 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:28 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:28.985+0000 7fbf18356200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:18:29.348 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:29 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:29.172+0000 7fbf18356200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:18:29.724 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:29 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:29.460+0000 7fbf18356200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:18:29.724 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:29 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:29.722+0000 7fbf18356200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:18:30.098 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:29 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:29.964+0000 7fbf18356200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:18:31.098 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:30 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:30.807+0000 7fbf18356200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:18:31.501 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:31 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/997936002' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:18:31.501 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:31 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:31.398+0000 7fbf18356200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:18:31.502 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:31 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:31.499+0000 7fbf18356200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:18:31.848 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:31 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:31.719+0000 7fbf18356200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:18:32.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:32 smithi007 ceph-mon[25800]: Activating manager daemon y 2024-03-12T03:18:32.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:32 smithi007 ceph-mon[25800]: mgrmap e2: y(active, starting, since 0.00302324s) 2024-03-12T03:18:32.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:32 smithi007 ceph-mon[25800]: from='mgr.14100 172.21.15.7:0/342976716' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:18:32.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:32 smithi007 ceph-mon[25800]: from='mgr.14100 172.21.15.7:0/342976716' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:18:32.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:32 smithi007 ceph-mon[25800]: from='mgr.14100 172.21.15.7:0/342976716' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:18:32.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:32 smithi007 ceph-mon[25800]: from='mgr.14100 172.21.15.7:0/342976716' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:18:32.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:32 smithi007 ceph-mon[25800]: from='mgr.14100 172.21.15.7:0/342976716' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:18:32.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:32 smithi007 ceph-mon[25800]: Manager daemon y is now available 2024-03-12T03:18:32.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:32 smithi007 ceph-mon[25800]: from='mgr.14100 172.21.15.7:0/342976716' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:18:32.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:32 smithi007 ceph-mon[25800]: from='mgr.14100 172.21.15.7:0/342976716' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:18:32.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:32 smithi007 ceph-mon[25800]: from='mgr.14100 172.21.15.7:0/342976716' entity='mgr.y' 2024-03-12T03:18:32.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:32 smithi007 ceph-mon[25800]: from='mgr.14100 172.21.15.7:0/342976716' entity='mgr.y' 2024-03-12T03:18:32.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:32 smithi007 ceph-mon[25800]: from='mgr.14100 172.21.15.7:0/342976716' entity='mgr.y' 2024-03-12T03:18:34.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:33 smithi007 ceph-mon[25800]: mgrmap e3: y(active, since 1.007s) 2024-03-12T03:18:36.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:35 smithi007 ceph-mon[25800]: mgrmap e4: y(active, since 3s) 2024-03-12T03:18:37.285 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-12T03:18:37.285 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:18:37.285 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "fe67b1e4-e01e-11ee-95c8-87774f69a715", 2024-03-12T03:18:37.285 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T03:18:37.285 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T03:18:37.285 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T03:18:37.285 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T03:18:37.286 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:37.286 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T03:18:37.286 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T03:18:37.286 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-03-12T03:18:37.286 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:18:37.286 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T03:18:37.286 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T03:18:37.286 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:18:37.286 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-12T03:18:37.286 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T03:18:37.286 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:18:37.287 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T03:18:37.287 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T03:18:37.287 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:37.287 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T03:18:37.287 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:18:37.287 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T03:18:37.287 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T03:18:37.287 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T03:18:37.290 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T03:18:37.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T03:18:37.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T03:18:37.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:37.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T03:18:37.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T03:18:37.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T03:18:37.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T03:18:37.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T03:18:37.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T03:18:37.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T03:18:37.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T03:18:37.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T03:18:37.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:37.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T03:18:37.291 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:18:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T03:18:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T03:18:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T03:18:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T03:18:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T03:18:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T03:18:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T03:18:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T03:18:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T03:18:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:18:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:18:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T03:18:37.292 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:18:37.293 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T03:18:17.656503+0000", 2024-03-12T03:18:37.293 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:18:37.293 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:37.293 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T03:18:37.293 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:18:37.293 INFO:teuthology.orchestra.run.smithi007.stdout:mgr not available, waiting (2/15)... 2024-03-12T03:18:40.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:40 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/2845084263' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:18:40.599 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-12T03:18:40.599 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:18:40.599 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "fe67b1e4-e01e-11ee-95c8-87774f69a715", 2024-03-12T03:18:40.599 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T03:18:40.599 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T03:18:40.599 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T03:18:40.599 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T03:18:40.599 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:40.599 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T03:18:40.599 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T03:18:40.600 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-03-12T03:18:40.600 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:18:40.600 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T03:18:40.600 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T03:18:40.600 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:18:40.600 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2024-03-12T03:18:40.600 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T03:18:40.600 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:18:40.600 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T03:18:40.600 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T03:18:40.600 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:40.600 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T03:18:40.600 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:18:40.600 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T03:18:40.600 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T03:18:40.601 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T03:18:40.601 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T03:18:40.601 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T03:18:40.601 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T03:18:40.601 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:40.601 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T03:18:40.601 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T03:18:40.604 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T03:18:40.604 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T03:18:40.604 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T03:18:40.604 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T03:18:40.604 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T03:18:40.604 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T03:18:40.604 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T03:18:40.604 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:40.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T03:18:40.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:18:40.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T03:18:40.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T03:18:40.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:40.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T03:18:40.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T03:18:40.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T03:18:40.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T03:18:40.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T03:18:40.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T03:18:40.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T03:18:40.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:18:40.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:18:40.605 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:40.606 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T03:18:40.606 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:18:40.606 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T03:18:17.656503+0000", 2024-03-12T03:18:40.606 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:18:40.606 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:18:40.606 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T03:18:40.606 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:18:40.606 INFO:teuthology.orchestra.run.smithi007.stdout:mgr is available 2024-03-12T03:18:41.540 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:41 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/379626804' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-12T03:18:41.540 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:41 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/379626804' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-12T03:18:41.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-12T03:18:41.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [global] 2024-03-12T03:18:41.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout fsid = fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:18:41.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-12T03:18:41.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-12T03:18:41.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-12T03:18:41.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-12T03:18:41.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-12T03:18:41.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-12T03:18:41.753 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-12T03:18:41.754 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-12T03:18:41.754 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [osd] 2024-03-12T03:18:41.754 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-12T03:18:41.754 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-12T03:18:41.754 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling cephadm module... 2024-03-12T03:18:42.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:42 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/3673974062' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-12T03:18:42.848 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:42 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: ignoring --setuser ceph since I am not root 2024-03-12T03:18:42.848 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:42 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: ignoring --setgroup ceph since I am not root 2024-03-12T03:18:43.438 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:43 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:43.138+0000 7ffab4a89200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:18:43.439 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:43 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:43.233+0000 7ffab4a89200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:18:43.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:43 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/3673974062' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-12T03:18:43.795 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:43 smithi007 ceph-mon[25800]: mgrmap e5: y(active, since 10s) 2024-03-12T03:18:44.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:43 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:43.905+0000 7ffab4a89200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:18:44.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:44 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:44.002+0000 7ffab4a89200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:18:44.338 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:44 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:44.148+0000 7ffab4a89200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:18:44.338 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:44 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:44.251+0000 7ffab4a89200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:18:44.339 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:44 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:44.336+0000 7ffab4a89200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:18:44.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:44 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/824263106' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T03:18:44.912 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:44 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:44.607+0000 7ffab4a89200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:18:44.912 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:44 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:44.724+0000 7ffab4a89200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:18:44.912 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:44 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:44.812+0000 7ffab4a89200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:18:45.107 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:18:45.107 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-12T03:18:45.107 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T03:18:45.107 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-12T03:18:45.107 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-12T03:18:45.107 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:18:45.107 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for the mgr to restart... 2024-03-12T03:18:45.107 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr epoch 5... 2024-03-12T03:18:45.212 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:44 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:44.909+0000 7ffab4a89200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:18:45.213 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:45 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:45.088+0000 7ffab4a89200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:18:46.348 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:46 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:46.082+0000 7ffab4a89200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:18:46.348 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:46 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:46.215+0000 7ffab4a89200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:18:46.848 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:46 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:46.560+0000 7ffab4a89200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:18:46.848 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:46 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:46.652+0000 7ffab4a89200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:18:47.330 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:46 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:46.850+0000 7ffab4a89200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:18:47.330 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:47 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:47.038+0000 7ffab4a89200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:18:47.591 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:47 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:47.328+0000 7ffab4a89200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:18:47.591 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:47 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:47.588+0000 7ffab4a89200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:18:47.848 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:47 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:47.827+0000 7ffab4a89200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:18:49.098 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:48 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:48.651+0000 7ffab4a89200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:18:49.555 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:49.232+0000 7ffab4a89200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:18:49.555 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:49.332+0000 7ffab4a89200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:18:49.555 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:18:49.552+0000 7ffab4a89200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:18:49.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-mon[25800]: Active manager daemon y restarted 2024-03-12T03:18:49.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-mon[25800]: Activating manager daemon y 2024-03-12T03:18:49.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-mon[25800]: osdmap e2: 0 total, 0 up, 0 in 2024-03-12T03:18:49.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-mon[25800]: mgrmap e6: y(active, starting, since 0.00317586s) 2024-03-12T03:18:49.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:18:49.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:18:49.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:18:49.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:18:49.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:18:49.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-mon[25800]: Manager daemon y is now available 2024-03-12T03:18:49.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:18:49.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:18:49.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:18:49.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:18:49.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:49 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:18:50.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:50 smithi007 ceph-mon[25800]: Found migration_current of "None". Setting to last migration. 2024-03-12T03:18:50.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:50 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:18:50.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:50 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:18:50.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:50 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:18:50.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:50 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:18:50.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:50 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:18:50.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:50 smithi007 ceph-mon[25800]: mgrmap e7: y(active, since 1.00493s) 2024-03-12T03:18:51.163 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:18:51.163 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-12T03:18:51.163 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-12T03:18:51.163 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:18:51.163 INFO:teuthology.orchestra.run.smithi007.stdout:mgr epoch 5 is available 2024-03-12T03:18:51.163 INFO:teuthology.orchestra.run.smithi007.stdout:Setting orchestrator backend to cephadm... 2024-03-12T03:18:51.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:51 smithi007 ceph-mon[25800]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T03:18:51.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:51 smithi007 ceph-mon[25800]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T03:18:52.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:52 smithi007 ceph-mon[25800]: mgrmap e8: y(active, since 2s) 2024-03-12T03:18:52.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:52 smithi007 ceph-mon[25800]: [12/Mar/2024:03:18:51] ENGINE Bus STARTING 2024-03-12T03:18:52.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:52 smithi007 ceph-mon[25800]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:18:52.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:52 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:18:52.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:52 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:18:52.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:52 smithi007 ceph-mon[25800]: [12/Mar/2024:03:18:51] ENGINE Serving on http://172.21.15.7:8765 2024-03-12T03:18:52.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:52 smithi007 ceph-mon[25800]: [12/Mar/2024:03:18:52] ENGINE Serving on https://172.21.15.7:7150 2024-03-12T03:18:52.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:52 smithi007 ceph-mon[25800]: [12/Mar/2024:03:18:52] ENGINE Bus STARTED 2024-03-12T03:18:52.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:52 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:18:53.654 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-12T03:18:53.654 INFO:teuthology.orchestra.run.smithi007.stdout:Generating ssh key... 2024-03-12T03:18:53.958 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:53 smithi007 ceph-mon[25800]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:18:54.762 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: Generating public/private rsa key pair. 2024-03-12T03:18:54.762 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: Your identification has been saved in /tmp/tmpihj11x10/key. 2024-03-12T03:18:54.762 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: Your public key has been saved in /tmp/tmpihj11x10/key.pub. 2024-03-12T03:18:54.762 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: The key fingerprint is: 2024-03-12T03:18:54.762 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: SHA256:BWEwlChAhGNxUUZcoTrTaF9UJIDIdFJCLm/uc+wcYqU ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:18:54.762 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: The key's randomart image is: 2024-03-12T03:18:54.762 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: +---[RSA 3072]----+ 2024-03-12T03:18:54.762 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: |OX+=*B***+ | 2024-03-12T03:18:54.762 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: |++*.o.ooo. | 2024-03-12T03:18:54.762 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: |oo . . . . | 2024-03-12T03:18:54.763 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: |.. + . . | 2024-03-12T03:18:54.763 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: | o*.. .S | 2024-03-12T03:18:54.763 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: | o.o+ . | 2024-03-12T03:18:54.763 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: | E... | 2024-03-12T03:18:54.763 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: | o.oo. | 2024-03-12T03:18:54.763 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: | .+o | 2024-03-12T03:18:54.763 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:18:54 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: +----[SHA256]-----+ 2024-03-12T03:18:55.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:55 smithi007 ceph-mon[25800]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:18:55.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:55 smithi007 ceph-mon[25800]: Generating ssh key... 2024-03-12T03:18:55.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:55 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:18:55.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:55 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:18:56.187 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5jp0No9lFN/RB1OU8UJqU4mvrEiyafmqHutOzGA0YsRkHnl6yhDmI7mobBWh5XQQCRNDnDbY/daODGLvWvXHu8FUOftZqWoTqzbIObGsA4ffay/512L396ElSPFr5SCKQKG4bt1xmbrONu3eS+oANA10QV1lFEzIXsOqTwlCOCjMQFfoZiOamdzwZOF2OahkG5vrrPfY4cCBGMGTU9I6WlW0KDSHIwfpTjiawz0pZkJM+myu/N0RTRM7x2wMSsgCJHb70Nlnfb4JH583s2mkagZuMPQXehS0GzkVnoWpB4VKde2Q/kTzhSZy6kQ3GwXomuiSn6U7+vGF3OVh587oH34l+4SOgLpN7SzwoaZAXGjJ40wuh8Sy9knKr4eo8JdcE/JE7cKHGKvzY9YTYy3B5nv733QZA+5aqofjvt4nMG26AZC8gb2Rf0WG9Dk76CxMCYeWNanNNtR++E8sdjD3tFnkj9jMxllBpcr3eX/ymaCPGgIQcfRSn2MIfm/6HUB0= ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:18:56.187 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-12T03:18:56.187 INFO:teuthology.orchestra.run.smithi007.stdout:Adding key to root@localhost authorized_keys... 2024-03-12T03:18:56.187 INFO:teuthology.orchestra.run.smithi007.stdout:Adding host smithi007... 2024-03-12T03:18:56.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:56 smithi007 ceph-mon[25800]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:18:57.913 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:57 smithi007 ceph-mon[25800]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi007", "addr": "172.21.15.7", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:18:57.913 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:18:57 smithi007 ceph-mon[25800]: Deploying cephadm binary to smithi007 2024-03-12T03:19:01.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Added host 'smithi007' with addr '172.21.15.7' 2024-03-12T03:19:01.861 INFO:teuthology.orchestra.run.smithi007.stdout:Deploying unmanaged mon service... 2024-03-12T03:19:02.557 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:02 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:19:02.557 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:02 smithi007 ceph-mon[25800]: Added host smithi007 2024-03-12T03:19:02.557 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:02 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:03.103 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-12T03:19:03.103 INFO:teuthology.orchestra.run.smithi007.stdout:Deploying unmanaged mgr service... 2024-03-12T03:19:03.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:03 smithi007 ceph-mon[25800]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:19:03.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:03 smithi007 ceph-mon[25800]: Saving service mon spec with placement count:5 2024-03-12T03:19:03.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:03 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:19:04.680 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-12T03:19:05.343 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:04 smithi007 ceph-mon[25800]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:19:05.343 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:04 smithi007 ceph-mon[25800]: Saving service mgr spec with placement count:2 2024-03-12T03:19:05.343 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:04 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:19:06.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:06 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:19:06.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:06 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:19:06.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:06 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/2834132895' entity='client.admin' 2024-03-12T03:19:07.754 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling the dashboard module... 2024-03-12T03:19:08.139 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:08 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/3670691823' entity='client.admin' 2024-03-12T03:19:09.426 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:09 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: ignoring --setuser ceph since I am not root 2024-03-12T03:19:09.427 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:09 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: ignoring --setgroup ceph since I am not root 2024-03-12T03:19:09.427 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:09 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/2676802505' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-12T03:19:09.427 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:09 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:19:09.427 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:09 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:19:09.427 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:09 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:19:09.427 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:09 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' 2024-03-12T03:19:09.427 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:09 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi007", "caps": []}]: dispatch 2024-03-12T03:19:09.427 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:09 smithi007 ceph-mon[25800]: from='mgr.14118 172.21.15.7:0/3801286372' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi007", "caps": []}]': finished 2024-03-12T03:19:09.944 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:09 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:09.688+0000 7fbdf42f6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:19:09.944 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:09 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:09.792+0000 7fbdf42f6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:19:10.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:10 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/2676802505' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-12T03:19:10.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:10 smithi007 ceph-mon[25800]: mgrmap e9: y(active, since 19s) 2024-03-12T03:19:10.708 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:10 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:10.464+0000 7fbdf42f6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:19:10.709 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:10 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:10.559+0000 7fbdf42f6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:19:10.709 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:10 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:10.706+0000 7fbdf42f6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:19:11.016 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:10 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:10.805+0000 7fbdf42f6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:19:11.017 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:10 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:10.890+0000 7fbdf42f6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:19:11.020 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:19:11.020 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-12T03:19:11.020 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T03:19:11.020 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-12T03:19:11.020 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-12T03:19:11.020 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:19:11.020 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for the mgr to restart... 2024-03-12T03:19:11.020 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr epoch 9... 2024-03-12T03:19:11.277 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:11 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/2802034161' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T03:19:11.277 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:11 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:11.166+0000 7fbdf42f6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:19:11.598 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:11 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:11.281+0000 7fbdf42f6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:19:11.598 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:11 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:11.372+0000 7fbdf42f6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:19:11.598 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:11 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:11.471+0000 7fbdf42f6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:19:12.098 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:11 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:11.650+0000 7fbdf42f6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:19:13.098 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:12 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:12.637+0000 7fbdf42f6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:19:13.098 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:12 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:12.771+0000 7fbdf42f6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:19:13.403 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:13 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:13.108+0000 7fbdf42f6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:19:13.404 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:13 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:13.201+0000 7fbdf42f6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:19:13.404 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:13 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:13.401+0000 7fbdf42f6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:19:13.848 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:13 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:13.590+0000 7fbdf42f6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:19:14.145 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:13 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:13.881+0000 7fbdf42f6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:19:14.145 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:14 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:14.142+0000 7fbdf42f6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:19:14.598 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:14 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:14.386+0000 7fbdf42f6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:19:15.598 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:15 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:15.212+0000 7fbdf42f6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:19:16.098 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:15 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:15.793+0000 7fbdf42f6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:19:16.098 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:15 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:15.893+0000 7fbdf42f6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:19:16.598 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:19:16 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:19:16.115+0000 7fbdf42f6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:19:16.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:16 smithi007 ceph-mon[25800]: Active manager daemon y restarted 2024-03-12T03:19:16.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:16 smithi007 ceph-mon[25800]: Activating manager daemon y 2024-03-12T03:19:16.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:16 smithi007 ceph-mon[25800]: osdmap e3: 0 total, 0 up, 0 in 2024-03-12T03:19:16.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:16 smithi007 ceph-mon[25800]: mgrmap e10: y(active, starting, since 0.00333353s) 2024-03-12T03:19:16.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:19:16.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:19:16.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:19:16.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:19:16.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:19:16.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:16 smithi007 ceph-mon[25800]: Manager daemon y is now available 2024-03-12T03:19:16.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:16.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:16.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:17.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:19:17.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:19:17.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:17 smithi007 ceph-mon[25800]: [12/Mar/2024:03:19:16] ENGINE Bus STARTING 2024-03-12T03:19:17.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:17 smithi007 ceph-mon[25800]: [12/Mar/2024:03:19:16] ENGINE Serving on http://172.21.15.7:8765 2024-03-12T03:19:17.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:17 smithi007 ceph-mon[25800]: [12/Mar/2024:03:19:16] ENGINE Serving on https://172.21.15.7:7150 2024-03-12T03:19:17.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:17 smithi007 ceph-mon[25800]: [12/Mar/2024:03:19:16] ENGINE Bus STARTED 2024-03-12T03:19:17.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:17 smithi007 ceph-mon[25800]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T03:19:17.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:17 smithi007 ceph-mon[25800]: mgrmap e11: y(active, since 1.00627s) 2024-03-12T03:19:17.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:17 smithi007 ceph-mon[25800]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T03:19:17.702 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:19:17.702 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-12T03:19:17.702 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-12T03:19:17.703 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:19:17.703 INFO:teuthology.orchestra.run.smithi007.stdout:mgr epoch 9 is available 2024-03-12T03:19:17.703 INFO:teuthology.orchestra.run.smithi007.stdout:Generating a dashboard self-signed certificate... 2024-03-12T03:19:18.993 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-12T03:19:18.993 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial admin user... 2024-03-12T03:19:19.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:19 smithi007 ceph-mon[25800]: mgrmap e12: y(active, since 2s) 2024-03-12T03:19:19.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:19 smithi007 ceph-mon[25800]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:19:19.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:19.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:19.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:19.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:19:19.369 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:19.370 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi007", "caps": []}]: dispatch 2024-03-12T03:19:19.370 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:19 smithi007 ceph-mon[25800]: Deploying daemon agent.smithi007 on smithi007 2024-03-12T03:19:20.584 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$0gYrIMBOnA74KcewmzTqie1gz75NKzR3abAlOBCpXnGPwt5JmTIJm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710213559, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-12T03:19:20.584 INFO:teuthology.orchestra.run.smithi007.stdout:Fetching dashboard port number... 2024-03-12T03:19:21.264 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:20 smithi007 ceph-mon[25800]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:19:21.265 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:20 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:21.864 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 8443 2024-03-12T03:19:21.873 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:19:21.873 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-03-12T03:19:21.881 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:19:21.881 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-03-12T03:19:21.882 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-03-12T03:19:21.882 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-12T03:19:21.882 INFO:teuthology.orchestra.run.smithi007.stdout:Ceph Dashboard is now available at: 2024-03-12T03:19:21.882 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:19:21.882 INFO:teuthology.orchestra.run.smithi007.stdout: URL: https://smithi007.front.sepia.ceph.com:8443/ 2024-03-12T03:19:21.882 INFO:teuthology.orchestra.run.smithi007.stdout: User: admin 2024-03-12T03:19:21.883 INFO:teuthology.orchestra.run.smithi007.stdout: Password: qh6ewxkxg9 2024-03-12T03:19:21.883 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:19:21.883 INFO:teuthology.orchestra.run.smithi007.stdout:Saving cluster configuration to /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config directory 2024-03-12T03:19:21.883 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling autotune for osd_memory_target 2024-03-12T03:19:22.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:21 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/1843814124' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-12T03:19:24.149 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:23 smithi007 ceph-mon[25800]: mgrmap e13: y(active, since 6s) 2024-03-12T03:19:24.149 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:23 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/1591917841' entity='client.admin' 2024-03-12T03:19:24.394 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-12T03:19:24.394 INFO:teuthology.orchestra.run.smithi007.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-12T03:19:24.394 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:19:24.394 INFO:teuthology.orchestra.run.smithi007.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:19:24.394 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:19:24.394 INFO:teuthology.orchestra.run.smithi007.stdout:Or, if you are only running a single cluster on this host: 2024-03-12T03:19:24.394 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:19:24.394 INFO:teuthology.orchestra.run.smithi007.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-12T03:19:24.394 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:19:24.394 INFO:teuthology.orchestra.run.smithi007.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-12T03:19:24.394 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:19:24.395 INFO:teuthology.orchestra.run.smithi007.stdout: ceph telemetry on 2024-03-12T03:19:24.395 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:19:24.395 INFO:teuthology.orchestra.run.smithi007.stdout:For more information see: 2024-03-12T03:19:24.395 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:19:24.395 INFO:teuthology.orchestra.run.smithi007.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-12T03:19:24.395 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:19:24.395 INFO:teuthology.orchestra.run.smithi007.stdout:Bootstrap complete. 2024-03-12T03:19:24.439 INFO:tasks.cephadm:Fetching config... 2024-03-12T03:19:24.440 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-12T03:19:24.440 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-12T03:19:24.457 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-12T03:19:24.457 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-12T03:19:24.457 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-12T03:19:24.513 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-12T03:19:24.514 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-12T03:19:24.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-12T03:19:24.700 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-12T03:19:24.700 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-12T03:19:24.700 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-12T03:19:24.754 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-12T03:19:24.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5jp0No9lFN/RB1OU8UJqU4mvrEiyafmqHutOzGA0YsRkHnl6yhDmI7mobBWh5XQQCRNDnDbY/daODGLvWvXHu8FUOftZqWoTqzbIObGsA4ffay/512L396ElSPFr5SCKQKG4bt1xmbrONu3eS+oANA10QV1lFEzIXsOqTwlCOCjMQFfoZiOamdzwZOF2OahkG5vrrPfY4cCBGMGTU9I6WlW0KDSHIwfpTjiawz0pZkJM+myu/N0RTRM7x2wMSsgCJHb70Nlnfb4JH583s2mkagZuMPQXehS0GzkVnoWpB4VKde2Q/kTzhSZy6kQ3GwXomuiSn6U7+vGF3OVh587oH34l+4SOgLpN7SzwoaZAXGjJ40wuh8Sy9knKr4eo8JdcE/JE7cKHGKvzY9YTYy3B5nv733QZA+5aqofjvt4nMG26AZC8gb2Rf0WG9Dk76CxMCYeWNanNNtR++E8sdjD3tFnkj9jMxllBpcr3eX/ymaCPGgIQcfRSn2MIfm/6HUB0= ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T03:19:24.920 INFO:teuthology.orchestra.run.smithi007.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5jp0No9lFN/RB1OU8UJqU4mvrEiyafmqHutOzGA0YsRkHnl6yhDmI7mobBWh5XQQCRNDnDbY/daODGLvWvXHu8FUOftZqWoTqzbIObGsA4ffay/512L396ElSPFr5SCKQKG4bt1xmbrONu3eS+oANA10QV1lFEzIXsOqTwlCOCjMQFfoZiOamdzwZOF2OahkG5vrrPfY4cCBGMGTU9I6WlW0KDSHIwfpTjiawz0pZkJM+myu/N0RTRM7x2wMSsgCJHb70Nlnfb4JH583s2mkagZuMPQXehS0GzkVnoWpB4VKde2Q/kTzhSZy6kQ3GwXomuiSn6U7+vGF3OVh587oH34l+4SOgLpN7SzwoaZAXGjJ40wuh8Sy9knKr4eo8JdcE/JE7cKHGKvzY9YTYy3B5nv733QZA+5aqofjvt4nMG26AZC8gb2Rf0WG9Dk76CxMCYeWNanNNtR++E8sdjD3tFnkj9jMxllBpcr3eX/ymaCPGgIQcfRSn2MIfm/6HUB0= ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:19:24.937 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5jp0No9lFN/RB1OU8UJqU4mvrEiyafmqHutOzGA0YsRkHnl6yhDmI7mobBWh5XQQCRNDnDbY/daODGLvWvXHu8FUOftZqWoTqzbIObGsA4ffay/512L396ElSPFr5SCKQKG4bt1xmbrONu3eS+oANA10QV1lFEzIXsOqTwlCOCjMQFfoZiOamdzwZOF2OahkG5vrrPfY4cCBGMGTU9I6WlW0KDSHIwfpTjiawz0pZkJM+myu/N0RTRM7x2wMSsgCJHb70Nlnfb4JH583s2mkagZuMPQXehS0GzkVnoWpB4VKde2Q/kTzhSZy6kQ3GwXomuiSn6U7+vGF3OVh587oH34l+4SOgLpN7SzwoaZAXGjJ40wuh8Sy9knKr4eo8JdcE/JE7cKHGKvzY9YTYy3B5nv733QZA+5aqofjvt4nMG26AZC8gb2Rf0WG9Dk76CxMCYeWNanNNtR++E8sdjD3tFnkj9jMxllBpcr3eX/ymaCPGgIQcfRSn2MIfm/6HUB0= ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T03:19:24.982 INFO:teuthology.orchestra.run.smithi033.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5jp0No9lFN/RB1OU8UJqU4mvrEiyafmqHutOzGA0YsRkHnl6yhDmI7mobBWh5XQQCRNDnDbY/daODGLvWvXHu8FUOftZqWoTqzbIObGsA4ffay/512L396ElSPFr5SCKQKG4bt1xmbrONu3eS+oANA10QV1lFEzIXsOqTwlCOCjMQFfoZiOamdzwZOF2OahkG5vrrPfY4cCBGMGTU9I6WlW0KDSHIwfpTjiawz0pZkJM+myu/N0RTRM7x2wMSsgCJHb70Nlnfb4JH583s2mkagZuMPQXehS0GzkVnoWpB4VKde2Q/kTzhSZy6kQ3GwXomuiSn6U7+vGF3OVh587oH34l+4SOgLpN7SzwoaZAXGjJ40wuh8Sy9knKr4eo8JdcE/JE7cKHGKvzY9YTYy3B5nv733QZA+5aqofjvt4nMG26AZC8gb2Rf0WG9Dk76CxMCYeWNanNNtR++E8sdjD3tFnkj9jMxllBpcr3eX/ymaCPGgIQcfRSn2MIfm/6HUB0= ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:19:24.997 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5jp0No9lFN/RB1OU8UJqU4mvrEiyafmqHutOzGA0YsRkHnl6yhDmI7mobBWh5XQQCRNDnDbY/daODGLvWvXHu8FUOftZqWoTqzbIObGsA4ffay/512L396ElSPFr5SCKQKG4bt1xmbrONu3eS+oANA10QV1lFEzIXsOqTwlCOCjMQFfoZiOamdzwZOF2OahkG5vrrPfY4cCBGMGTU9I6WlW0KDSHIwfpTjiawz0pZkJM+myu/N0RTRM7x2wMSsgCJHb70Nlnfb4JH583s2mkagZuMPQXehS0GzkVnoWpB4VKde2Q/kTzhSZy6kQ3GwXomuiSn6U7+vGF3OVh587oH34l+4SOgLpN7SzwoaZAXGjJ40wuh8Sy9knKr4eo8JdcE/JE7cKHGKvzY9YTYy3B5nv733QZA+5aqofjvt4nMG26AZC8gb2Rf0WG9Dk76CxMCYeWNanNNtR++E8sdjD3tFnkj9jMxllBpcr3eX/ymaCPGgIQcfRSn2MIfm/6HUB0= ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T03:19:25.041 INFO:teuthology.orchestra.run.smithi170.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5jp0No9lFN/RB1OU8UJqU4mvrEiyafmqHutOzGA0YsRkHnl6yhDmI7mobBWh5XQQCRNDnDbY/daODGLvWvXHu8FUOftZqWoTqzbIObGsA4ffay/512L396ElSPFr5SCKQKG4bt1xmbrONu3eS+oANA10QV1lFEzIXsOqTwlCOCjMQFfoZiOamdzwZOF2OahkG5vrrPfY4cCBGMGTU9I6WlW0KDSHIwfpTjiawz0pZkJM+myu/N0RTRM7x2wMSsgCJHb70Nlnfb4JH583s2mkagZuMPQXehS0GzkVnoWpB4VKde2Q/kTzhSZy6kQ3GwXomuiSn6U7+vGF3OVh587oH34l+4SOgLpN7SzwoaZAXGjJ40wuh8Sy9knKr4eo8JdcE/JE7cKHGKvzY9YTYy3B5nv733QZA+5aqofjvt4nMG26AZC8gb2Rf0WG9Dk76CxMCYeWNanNNtR++E8sdjD3tFnkj9jMxllBpcr3eX/ymaCPGgIQcfRSn2MIfm/6HUB0= ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:19:25.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-12T03:19:25.314 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:19:26.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:26.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:26.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:26.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:26.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:26.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:26.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:26.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:26.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:26.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:26.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:26.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:27.270 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:27 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:28.008 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-12T03:19:28.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-12T03:19:28.826 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:19:28.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:28 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/1490977915' entity='client.admin' 2024-03-12T03:19:28.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:28 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:28.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:28 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:30.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:30.373 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:32.167 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi033 2024-03-12T03:19:32.168 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-12T03:19:32.168 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/etc/ceph/ceph.conf 2024-03-12T03:19:32.185 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-12T03:19:32.185 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:19:32.242 INFO:tasks.cephadm:Adding host smithi033 to orchestrator... 2024-03-12T03:19:32.242 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph orch host add smithi033 2024-03-12T03:19:32.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:32 smithi007 ceph-mon[25800]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:19:32.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:32 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:32.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:32 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:32.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:32 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:32.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:32 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:32.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:32 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:32.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:32 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:32.664 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:19:33.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:33 smithi007 ceph-mon[25800]: Updating smithi007:/etc/ceph/ceph.conf 2024-03-12T03:19:33.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:33 smithi007 ceph-mon[25800]: Updating smithi007:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:19:33.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:33 smithi007 ceph-mon[25800]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:19:33.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:33 smithi007 ceph-mon[25800]: Updating smithi007:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:19:33.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:33 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:33.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:33 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:35.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:35 smithi007 ceph-mon[25800]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi033", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:19:35.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:35 smithi007 ceph-mon[25800]: Deploying cephadm binary to smithi033 2024-03-12T03:19:37.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:37 smithi007 ceph-mon[25800]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:38.864 INFO:teuthology.orchestra.run.smithi007.stdout:Added host 'smithi033' with addr '172.21.15.33' 2024-03-12T03:19:39.548 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:39 smithi007 ceph-mon[25800]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:39.548 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:39 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:39.549 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:39 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:39.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-12T03:19:39.827 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:19:40.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:40 smithi007 ceph-mon[25800]: Added host smithi033 2024-03-12T03:19:40.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:40 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:40.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:40 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:40.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:40 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:40.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:40 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:19:40.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:40 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:40.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:40 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:41.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:41 smithi007 ceph-mon[25800]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-12T03:19:41.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:41 smithi007 ceph-mon[25800]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:41.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:41 smithi007 ceph-mon[25800]: Updating smithi033:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:19:41.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:41 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:41.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:41 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:41.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:41 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi033", "caps": []}]: dispatch 2024-03-12T03:19:41.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:41 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi033", "caps": []}]': finished 2024-03-12T03:19:41.634 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:19:41.634 INFO:teuthology.orchestra.run.smithi007.stdout:[{"addr": "172.21.15.7", "hostname": "smithi007", "labels": [], "status": ""}, {"addr": "172.21.15.33", "hostname": "smithi033", "labels": [], "status": ""}] 2024-03-12T03:19:42.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:42 smithi007 ceph-mon[25800]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:19:42.178 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:42 smithi007 ceph-mon[25800]: Updating smithi033:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:19:42.204 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi170 2024-03-12T03:19:42.204 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-12T03:19:42.204 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.conf 2024-03-12T03:19:42.222 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-12T03:19:42.222 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:19:42.280 INFO:tasks.cephadm:Adding host smithi170 to orchestrator... 2024-03-12T03:19:42.280 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph orch host add smithi170 2024-03-12T03:19:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:19:43.310 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:43 smithi007 ceph-mon[25800]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:19:43.310 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:43 smithi007 ceph-mon[25800]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:44.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:44 smithi007 ceph-mon[25800]: Deploying daemon agent.smithi033 on smithi033 2024-03-12T03:19:44.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:44 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:44.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:44 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:44.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:44 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:44.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:44 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:44.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:44 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:44.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:44 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:44.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:44 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:44.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:44 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:44.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:44 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:44.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:44 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:44.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:44 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:44.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:44 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:44.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:44 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:44.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:44 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:45.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:45 smithi007 ceph-mon[25800]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:46.330 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:46 smithi007 ceph-mon[25800]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:19:46.330 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:46 smithi007 ceph-mon[25800]: Deploying cephadm binary to smithi170 2024-03-12T03:19:47.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:47 smithi007 ceph-mon[25800]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:47.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:47 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:47.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:47 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:47.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:47 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:47.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:47 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:47.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:47 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:47.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:47 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:48.617 INFO:teuthology.orchestra.run.smithi007.stdout:Added host 'smithi170' with addr '172.21.15.170' 2024-03-12T03:19:49.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:49 smithi007 ceph-mon[25800]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:49.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:49 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:49.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:49 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:49.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:49 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:49.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-12T03:19:49.468 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:19:50.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:50 smithi007 ceph-mon[25800]: Added host smithi170 2024-03-12T03:19:50.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:50 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:50.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:50 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:50.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:50 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:19:50.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:50 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:50.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:50 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:51.235 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:19:51.235 INFO:teuthology.orchestra.run.smithi007.stdout:[{"addr": "172.21.15.7", "hostname": "smithi007", "labels": [], "status": ""}, {"addr": "172.21.15.33", "hostname": "smithi033", "labels": [], "status": ""}, {"addr": "172.21.15.170", "hostname": "smithi170", "labels": [], "status": ""}] 2024-03-12T03:19:51.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:51 smithi007 ceph-mon[25800]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-12T03:19:51.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:51 smithi007 ceph-mon[25800]: Updating smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:19:51.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:51 smithi007 ceph-mon[25800]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:19:51.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:51 smithi007 ceph-mon[25800]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:51.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:51 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:51.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:51 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:51.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:51 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]: dispatch 2024-03-12T03:19:51.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:51 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]': finished 2024-03-12T03:19:51.893 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-12T03:19:51.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-12T03:19:52.128 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:19:52.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:52 smithi007 ceph-mon[25800]: Updating smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:19:52.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:52 smithi007 ceph-mon[25800]: Deploying daemon agent.smithi170 on smithi170 2024-03-12T03:19:52.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:52.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:52.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:52.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:52.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:52.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:52.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:52.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:52.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:52.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:52.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:52.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:53.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:53 smithi007 ceph-mon[25800]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:19:53.545 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:53 smithi007 ceph-mon[25800]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:53.545 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:53.545 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:54.166 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2024-03-12T03:19:54.422 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:54 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/1093224237' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T03:19:54.788 INFO:tasks.cephadm:Adding mon.a on smithi007 2024-03-12T03:19:54.788 INFO:tasks.cephadm:Adding mon.b on smithi033 2024-03-12T03:19:54.789 INFO:tasks.cephadm:Adding mon.c on smithi170 2024-03-12T03:19:54.789 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi007:172.21.15.7=a;smithi033:172.21.15.33=b;smithi170:172.21.15.170=c' 2024-03-12T03:19:55.024 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:19:55.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:55 smithi007 ceph-mon[25800]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:55.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:55 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/1093224237' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T03:19:55.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:55 smithi007 ceph-mon[25800]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:19:55.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:55 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:55.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:55 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:56.777 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled mon update... 2024-03-12T03:19:57.325 DEBUG:teuthology.orchestra.run.smithi033:mon.b> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mon.b.service 2024-03-12T03:19:57.328 DEBUG:teuthology.orchestra.run.smithi170:mon.c> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mon.c.service 2024-03-12T03:19:57.330 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T03:19:57.330 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:19:57.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:57 smithi007 ceph-mon[25800]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:57.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:57 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:57.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:57 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:57.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:57 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:57.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:57 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:57.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:57 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:57.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:57 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:19:57.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:57 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:19:57.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:57 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:58.114 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:19:58.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:58 smithi007 ceph-mon[25800]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi033:172.21.15.33=b;smithi170:172.21.15.170=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:19:58.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:58 smithi007 ceph-mon[25800]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi033:172.21.15.33=b;smithi170:172.21.15.170=c;count:3 2024-03-12T03:19:58.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:58 smithi007 ceph-mon[25800]: Deploying daemon mon.c on smithi170 2024-03-12T03:19:59.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:19:59 smithi007 ceph-mon[25800]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:00.571 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-12T03:20:00.571 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"fe67b1e4-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:18:14.661163Z","created":"2024-03-12T03:18:14.661163Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:20:00.573 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2024-03-12T03:20:01.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:01 smithi007 ceph-mon[25800]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:01.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:01 smithi007 ceph-mon[25800]: from='client.? 172.21.15.170:0/3968539329' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:20:02.629 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-12T03:20:02.629 INFO:journalctl@ceph.mon.c.smithi170.stdout: e1 2024-03-12T03:20:02.629 INFO:journalctl@ceph.mon.c.smithi170.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:20:02.629 INFO:journalctl@ceph.mon.c.smithi170.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T03:20:02.630 INFO:journalctl@ceph.mon.c.smithi170.stdout: legacy client fscid: -1 2024-03-12T03:20:02.630 INFO:journalctl@ceph.mon.c.smithi170.stdout: 2024-03-12T03:20:02.630 INFO:journalctl@ceph.mon.c.smithi170.stdout: No filesystems configured 2024-03-12T03:20:02.630 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-12T03:20:02.630 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-12T03:20:02.684 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T03:20:02.685 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:20:02.917 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T03:20:03.028 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-12T03:20:03.028 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-12T03:20:03.028 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-12T03:20:03.028 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-12T03:20:03.028 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T03:20:03.028 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:20:03.028 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:20:03.028 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:20:03.028 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:20:03.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:03.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: Deploying daemon agent.smithi033 on smithi033 2024-03-12T03:20:03.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:03.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:03.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:03.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:03.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:03.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:03.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:03.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:03.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: Deploying cephadm binary to smithi170 2024-03-12T03:20:03.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:03.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:03.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:03.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: Added host smithi170 2024-03-12T03:20:03.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:03.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:03.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:03.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-12T03:20:03.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: Updating smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:20:03.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:20:03.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:03.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.031 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]: dispatch 2024-03-12T03:20:03.032 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]': finished 2024-03-12T03:20:03.032 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: Updating smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:20:03.032 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: Deploying daemon agent.smithi170 on smithi170 2024-03-12T03:20:03.032 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.032 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.032 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.032 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:03.032 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:03.032 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:03.032 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.032 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:03.032 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:03.032 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:03.033 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.033 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.033 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:20:03.033 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:03.033 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.033 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.033 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/1093224237' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T03:20:03.033 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:03.033 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/1093224237' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T03:20:03.033 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:20:03.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:03.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:03.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:03.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:03.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:03.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:20:03.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:03.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi033:172.21.15.33=b;smithi170:172.21.15.170=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:03.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi033:172.21.15.33=b;smithi170:172.21.15.170=c;count:3 2024-03-12T03:20:03.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: Deploying daemon mon.c on smithi170 2024-03-12T03:20:03.034 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:03.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:03.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: from='client.? 172.21.15.170:0/3968539329' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:20:03.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 ceph-mon[27708]: mon.c@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-12T03:20:03.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 bash[27670]: a8d1056bc966b66b2aba5b4bd0d409cbdd062c555d8e2d37c3f6975a4c54bb21 2024-03-12T03:20:03.035 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:02 smithi170 systemd[1]: Started Ceph mon.c for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:20:08.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:20:08.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:08.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: mon.a calling monitor election 2024-03-12T03:20:08.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:08.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:08.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:08.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: mon.c calling monitor election 2024-03-12T03:20:08.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:08.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:08.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:08.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:08.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T03:20:08.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:20:08.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: fsmap 2024-03-12T03:20:08.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:20:08.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: mgrmap e13: y(active, since 51s) 2024-03-12T03:20:08.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: overall HEALTH_OK 2024-03-12T03:20:08.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:20:08.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:08.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:20:08.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:08.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: mon.a calling monitor election 2024-03-12T03:20:08.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:08.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:08.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:08.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: mon.c calling monitor election 2024-03-12T03:20:08.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:08.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:08.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:08.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:08.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T03:20:08.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:20:08.115 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: fsmap 2024-03-12T03:20:08.115 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:20:08.115 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: mgrmap e13: y(active, since 51s) 2024-03-12T03:20:08.115 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: overall HEALTH_OK 2024-03-12T03:20:08.115 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.115 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.115 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.115 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.115 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.115 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.115 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.115 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:20:08.115 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.115 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:08.116 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:08.189 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-12T03:20:08.189 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":2,"fsid":"fe67b1e4-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:20:02.634476Z","created":"2024-03-12T03:18:14.661163Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-12T03:20:08.191 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 2 2024-03-12T03:20:09.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:08 smithi007 ceph-mon[25800]: Deploying daemon mon.b on smithi033 2024-03-12T03:20:09.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:08 smithi007 ceph-mon[25800]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:09.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:08 smithi007 ceph-mon[25800]: from='client.? 172.21.15.170:0/514435170' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:20:09.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:08 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:09.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:08 smithi170 ceph-mon[27708]: Deploying daemon mon.b on smithi033 2024-03-12T03:20:09.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:08 smithi170 ceph-mon[27708]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:09.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:08 smithi170 ceph-mon[27708]: from='client.? 172.21.15.170:0/514435170' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:20:09.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:08 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:09.714 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T03:20:09.714 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:20:09.961 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T03:20:10.357 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:10 smithi170 ceph-mon[27708]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:10.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:10 smithi007 ceph-mon[25800]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:11.605 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 podman[27515]: 2024-03-12 03:20:11.357281779 +0000 UTC m=+0.017949787 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:20:11.605 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 podman[27515]: 2024-03-12 03:20:11.603517603 +0000 UTC m=+0.264185606 container init 8147033ac0829f010587c741e47ee8c08099300b5da14d3bc3ac478e13b422d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-b, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-12T03:20:12.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 podman[27515]: 2024-03-12 03:20:11.60900654 +0000 UTC m=+0.269674542 container start 8147033ac0829f010587c741e47ee8c08099300b5da14d3bc3ac478e13b422d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-12T03:20:12.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T03:20:12.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 7 2024-03-12T03:20:12.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: pidfile_write: ignore empty --pid-file 2024-03-12T03:20:12.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: load: jerasure load: lrc 2024-03-12T03:20:12.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: RocksDB version: 7.9.2 2024-03-12T03:20:12.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Git sha 0 2024-03-12T03:20:12.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-12T03:20:12.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: DB SUMMARY 2024-03-12T03:20:12.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: DB Session ID: 45J8MI0LWT88VHAJ67GJ 2024-03-12T03:20:12.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: CURRENT file: CURRENT 2024-03-12T03:20:12.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T03:20:12.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-12T03:20:12.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-12T03:20:12.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-12T03:20:12.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.error_if_exists: 0 2024-03-12T03:20:12.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.create_if_missing: 0 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.paranoid_checks: 1 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.env: 0x55c2d3974b20 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.info_log: 0x55c2d43318e0 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.statistics: (nil) 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.use_fsync: 0 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_log_file_size: 0 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T03:20:12.015 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.allow_fallocate: 1 2024-03-12T03:20:12.017 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T03:20:12.018 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T03:20:12.018 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.use_direct_reads: 0 2024-03-12T03:20:12.018 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T03:20:12.018 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T03:20:12.018 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.db_log_dir: 2024-03-12T03:20:12.018 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.wal_dir: 2024-03-12T03:20:12.018 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T03:20:12.018 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T03:20:12.018 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T03:20:12.018 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T03:20:12.018 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T03:20:12.018 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T03:20:12.018 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T03:20:12.019 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T03:20:12.019 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.write_buffer_manager: 0x55c2d4305400 2024-03-12T03:20:12.019 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T03:20:12.019 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T03:20:12.019 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T03:20:12.019 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T03:20:12.019 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T03:20:12.019 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T03:20:12.019 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T03:20:12.019 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T03:20:12.019 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.unordered_write: 0 2024-03-12T03:20:12.019 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T03:20:12.019 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T03:20:12.019 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T03:20:12.020 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T03:20:12.020 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.row_cache: None 2024-03-12T03:20:12.020 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.wal_filter: None 2024-03-12T03:20:12.020 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T03:20:12.020 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T03:20:12.020 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.two_write_queues: 0 2024-03-12T03:20:12.020 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T03:20:12.020 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.wal_compression: 0 2024-03-12T03:20:12.020 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.atomic_flush: 0 2024-03-12T03:20:12.020 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T03:20:12.020 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T03:20:12.020 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T03:20:12.020 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.log_readahead_size: 0 2024-03-12T03:20:12.020 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T03:20:12.020 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T03:20:12.021 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T03:20:12.021 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T03:20:12.021 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T03:20:12.021 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T03:20:12.021 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T03:20:12.021 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_background_jobs: 2 2024-03-12T03:20:12.021 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_background_compactions: -1 2024-03-12T03:20:12.021 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_subcompactions: 1 2024-03-12T03:20:12.021 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T03:20:12.021 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T03:20:12.021 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T03:20:12.021 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T03:20:12.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T03:20:12.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T03:20:12.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T03:20:12.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T03:20:12.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_open_files: -1 2024-03-12T03:20:12.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T03:20:12.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T03:20:12.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T03:20:12.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T03:20:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_background_flushes: -1 2024-03-12T03:20:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Compression algorithms supported: 2024-03-12T03:20:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T03:20:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: kZSTD supported: 0 2024-03-12T03:20:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: kXpressCompression supported: 0 2024-03-12T03:20:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T03:20:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: kZlibCompression supported: 1 2024-03-12T03:20:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: kSnappyCompression supported: 1 2024-03-12T03:20:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: kLZ4Compression supported: 1 2024-03-12T03:20:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: kBZip2Compression supported: 0 2024-03-12T03:20:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T03:20:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T03:20:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-12T03:20:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T03:20:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T03:20:12.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.merge_operator: 2024-03-12T03:20:12.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compaction_filter: None 2024-03-12T03:20:12.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T03:20:12.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T03:20:12.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T03:20:12.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T03:20:12.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c2d43319e0) 2024-03-12T03:20:12.024 INFO:journalctl@ceph.mon.b.smithi033.stdout: cache_index_and_filter_blocks: 1 2024-03-12T03:20:12.024 INFO:journalctl@ceph.mon.b.smithi033.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T03:20:12.024 INFO:journalctl@ceph.mon.b.smithi033.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T03:20:12.024 INFO:journalctl@ceph.mon.b.smithi033.stdout: pin_top_level_index_and_filter: 1 2024-03-12T03:20:12.024 INFO:journalctl@ceph.mon.b.smithi033.stdout: index_type: 0 2024-03-12T03:20:12.024 INFO:journalctl@ceph.mon.b.smithi033.stdout: data_block_index_type: 0 2024-03-12T03:20:12.024 INFO:journalctl@ceph.mon.b.smithi033.stdout: index_shortening: 1 2024-03-12T03:20:12.025 INFO:journalctl@ceph.mon.b.smithi033.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T03:20:12.025 INFO:journalctl@ceph.mon.b.smithi033.stdout: checksum: 4 2024-03-12T03:20:12.025 INFO:journalctl@ceph.mon.b.smithi033.stdout: no_block_cache: 0 2024-03-12T03:20:12.025 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_cache: 0x55c2d4337090 2024-03-12T03:20:12.025 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_cache_name: BinnedLRUCache 2024-03-12T03:20:12.025 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_cache_options: 2024-03-12T03:20:12.025 INFO:journalctl@ceph.mon.b.smithi033.stdout: capacity : 536870912 2024-03-12T03:20:12.025 INFO:journalctl@ceph.mon.b.smithi033.stdout: num_shard_bits : 4 2024-03-12T03:20:12.025 INFO:journalctl@ceph.mon.b.smithi033.stdout: strict_capacity_limit : 0 2024-03-12T03:20:12.025 INFO:journalctl@ceph.mon.b.smithi033.stdout: high_pri_pool_ratio: 0.000 2024-03-12T03:20:12.025 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_cache_compressed: (nil) 2024-03-12T03:20:12.025 INFO:journalctl@ceph.mon.b.smithi033.stdout: persistent_cache: (nil) 2024-03-12T03:20:12.025 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_size: 4096 2024-03-12T03:20:12.025 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_size_deviation: 10 2024-03-12T03:20:12.025 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_restart_interval: 16 2024-03-12T03:20:12.026 INFO:journalctl@ceph.mon.b.smithi033.stdout: index_block_restart_interval: 1 2024-03-12T03:20:12.026 INFO:journalctl@ceph.mon.b.smithi033.stdout: metadata_block_size: 4096 2024-03-12T03:20:12.026 INFO:journalctl@ceph.mon.b.smithi033.stdout: partition_filters: 0 2024-03-12T03:20:12.026 INFO:journalctl@ceph.mon.b.smithi033.stdout: use_delta_encoding: 1 2024-03-12T03:20:12.026 INFO:journalctl@ceph.mon.b.smithi033.stdout: filter_policy: bloomfilter 2024-03-12T03:20:12.026 INFO:journalctl@ceph.mon.b.smithi033.stdout: whole_key_filtering: 1 2024-03-12T03:20:12.026 INFO:journalctl@ceph.mon.b.smithi033.stdout: verify_compression: 0 2024-03-12T03:20:12.026 INFO:journalctl@ceph.mon.b.smithi033.stdout: read_amp_bytes_per_bit: 0 2024-03-12T03:20:12.026 INFO:journalctl@ceph.mon.b.smithi033.stdout: format_version: 5 2024-03-12T03:20:12.026 INFO:journalctl@ceph.mon.b.smithi033.stdout: enable_index_compression: 1 2024-03-12T03:20:12.026 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_align: 0 2024-03-12T03:20:12.026 INFO:journalctl@ceph.mon.b.smithi033.stdout: max_auto_readahead_size: 262144 2024-03-12T03:20:12.026 INFO:journalctl@ceph.mon.b.smithi033.stdout: prepopulate_block_cache: 0 2024-03-12T03:20:12.026 INFO:journalctl@ceph.mon.b.smithi033.stdout: initial_auto_readahead_size: 8192 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compression: NoCompression 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.num_levels: 7 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T03:20:12.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T03:20:12.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T03:20:12.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T03:20:12.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T03:20:12.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T03:20:12.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T03:20:12.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T03:20:12.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T03:20:12.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T03:20:12.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T03:20:12.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T03:20:12.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T03:20:12.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T03:20:12.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T03:20:12.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T03:20:12.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T03:20:12.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T03:20:12.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T03:20:12.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T03:20:12.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T03:20:12.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T03:20:12.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T03:20:12.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T03:20:12.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T03:20:12.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T03:20:12.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T03:20:12.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T03:20:12.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T03:20:12.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T03:20:12.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T03:20:12.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.table_properties_collectors: 2024-03-12T03:20:12.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.inplace_update_support: 0 2024-03-12T03:20:12.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T03:20:12.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T03:20:12.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T03:20:12.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T03:20:12.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.bloom_locality: 0 2024-03-12T03:20:12.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.max_successive_merges: 0 2024-03-12T03:20:12.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T03:20:12.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T03:20:12.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T03:20:12.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T03:20:12.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.ttl: 2592000 2024-03-12T03:20:12.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T03:20:12.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T03:20:12.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T03:20:12.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.enable_blob_files: false 2024-03-12T03:20:12.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.min_blob_size: 0 2024-03-12T03:20:12.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T03:20:12.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T03:20:12.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T03:20:12.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T03:20:12.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T03:20:12.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T03:20:12.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T03:20:12.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T03:20:12.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-12T03:20:12.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-12T03:20:12.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1dc8197c-9907-4240-9213-20cea7ee04d1 2024-03-12T03:20:12.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710213611639042, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-12T03:20:12.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-12T03:20:12.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710213611639471, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710213611, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1dc8197c-9907-4240-9213-20cea7ee04d1", "db_session_id": "45J8MI0LWT88VHAJ67GJ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-12T03:20:12.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710213611639554, "job": 1, "event": "recovery_finished"} 2024-03-12T03:20:12.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-12T03:20:12.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T03:20:12.034 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c2d442a000 2024-03-12T03:20:12.034 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: DB pointer 0x55c2d4414000 2024-03-12T03:20:12.034 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T03:20:12.034 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T03:20:12.034 INFO:journalctl@ceph.mon.b.smithi033.stdout: ** DB Stats ** 2024-03-12T03:20:12.034 INFO:journalctl@ceph.mon.b.smithi033.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:20:12.034 INFO:journalctl@ceph.mon.b.smithi033.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T03:20:12.034 INFO:journalctl@ceph.mon.b.smithi033.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:20:12.034 INFO:journalctl@ceph.mon.b.smithi033.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:20:12.034 INFO:journalctl@ceph.mon.b.smithi033.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T03:20:12.034 INFO:journalctl@ceph.mon.b.smithi033.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:20:12.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:20:12.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-03-12T03:20:12.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: ** Compaction Stats [default] ** 2024-03-12T03:20:12.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:20:12.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T03:20:12.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:20:12.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:20:12.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:20:12.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-03-12T03:20:12.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: ** Compaction Stats [default] ** 2024-03-12T03:20:12.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:20:12.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:20:12.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T03:20:12.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-03-12T03:20:12.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T03:20:12.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-03-12T03:20:12.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:20:12.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:20:12.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:20:12.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:20:12.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:20:12.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:20:12.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:20:12.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:20:12.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:20:12.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: Block cache BinnedLRUCache@0x55c2d4337090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-12T03:20:12.037 INFO:journalctl@ceph.mon.b.smithi033.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-12T03:20:12.037 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-03-12T03:20:12.037 INFO:journalctl@ceph.mon.b.smithi033.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:20:12.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-12T03:20:12.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: using public_addr v2:172.21.15.33:0/0 -> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] 2024-03-12T03:20:12.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: starting mon.b rank -1 at public addrs [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] at bind addrs [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:20:12.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.b@-1(???) e0 preinit fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:20:12.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.b@-1(synchronizing).mds e1 new map 2024-03-12T03:20:12.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-12T03:20:12.037 INFO:journalctl@ceph.mon.b.smithi033.stdout: e1 2024-03-12T03:20:12.037 INFO:journalctl@ceph.mon.b.smithi033.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:20:12.037 INFO:journalctl@ceph.mon.b.smithi033.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T03:20:12.037 INFO:journalctl@ceph.mon.b.smithi033.stdout: legacy client fscid: -1 2024-03-12T03:20:12.037 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-03-12T03:20:12.038 INFO:journalctl@ceph.mon.b.smithi033.stdout: No filesystems configured 2024-03-12T03:20:12.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-12T03:20:12.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-12T03:20:12.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-12T03:20:12.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-12T03:20:12.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-12T03:20:12.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-12T03:20:12.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T03:20:12.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:20:12.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:20:12.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:20:12.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:20:12.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:12.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: Deploying daemon agent.smithi033 on smithi033 2024-03-12T03:20:12.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:12.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:12.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:12.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:12.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:12.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:12.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:12.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:12.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: Deploying cephadm binary to smithi170 2024-03-12T03:20:12.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:12.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:12.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:12.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: Added host smithi170 2024-03-12T03:20:12.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:12.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:12.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:12.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-12T03:20:12.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: Updating smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:20:12.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:20:12.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:12.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]: dispatch 2024-03-12T03:20:12.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]': finished 2024-03-12T03:20:12.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: Updating smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:20:12.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: Deploying daemon agent.smithi170 on smithi170 2024-03-12T03:20:12.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:12.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:12.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:12.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:12.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:12.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:12.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:20:12.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:12.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.044 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/1093224237' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T03:20:12.044 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:12.044 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/1093224237' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T03:20:12.044 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:20:12.044 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.044 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.044 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:12.044 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.044 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.044 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:12.044 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:12.044 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:12.044 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.044 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:20:12.044 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi033:172.21.15.33=b;smithi170:172.21.15.170=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi033:172.21.15.33=b;smithi170:172.21.15.170=c;count:3 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: Deploying daemon mon.c on smithi170 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='client.? 172.21.15.170:0/3968539329' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.a calling monitor election 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.c calling monitor election 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:12.045 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:12.046 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:12.046 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T03:20:12.046 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:20:12.046 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: fsmap 2024-03-12T03:20:12.046 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:20:12.046 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mgrmap e13: y(active, since 51s) 2024-03-12T03:20:12.046 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: overall HEALTH_OK 2024-03-12T03:20:12.046 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.046 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.046 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.046 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.046 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.046 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:20:12.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:12.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:12.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: Deploying daemon mon.b on smithi033 2024-03-12T03:20:12.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:12.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='client.? 172.21.15.170:0/514435170' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:20:12.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:12.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:12.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 ceph-mon[27552]: mon.b@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-12T03:20:12.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 bash[27515]: 8147033ac0829f010587c741e47ee8c08099300b5da14d3bc3ac478e13b422d0 2024-03-12T03:20:12.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:11 smithi033 systemd[1]: Started Ceph mon.b for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:20:16.717 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-12T03:20:16.717 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":3,"fsid":"fe67b1e4-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:20:11.666837Z","created":"2024-03-12T03:18:14.661163Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-12T03:20:16.719 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 3 2024-03-12T03:20:17.049 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:20:17.049 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.049 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: mon.a calling monitor election 2024-03-12T03:20:17.050 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: mon.c calling monitor election 2024-03-12T03:20:17.050 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:17.050 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='client.? 172.21.15.170:0/408128779' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:20:17.050 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:17.050 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.050 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.050 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: mon.b calling monitor election 2024-03-12T03:20:17.050 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:17.050 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.050 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.050 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:17.050 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.050 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T03:20:17.050 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:20:17.051 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: fsmap 2024-03-12T03:20:17.051 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:20:17.051 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: mgrmap e13: y(active, since 60s) 2024-03-12T03:20:17.051 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: overall HEALTH_OK 2024-03-12T03:20:17.051 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.051 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.051 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.051 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.051 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.051 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.051 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: mon.a calling monitor election 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: mon.c calling monitor election 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='client.? 172.21.15.170:0/408128779' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: mon.b calling monitor election 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T03:20:17.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:20:17.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: fsmap 2024-03-12T03:20:17.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:20:17.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: mgrmap e13: y(active, since 60s) 2024-03-12T03:20:17.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: overall HEALTH_OK 2024-03-12T03:20:17.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.100 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:20:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: mon.a calling monitor election 2024-03-12T03:20:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: mon.c calling monitor election 2024-03-12T03:20:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:20:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='client.? 172.21.15.170:0/408128779' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:20:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:17.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: mon.b calling monitor election 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: fsmap 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: mgrmap e13: y(active, since 60s) 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: overall HEALTH_OK 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.265 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.385 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-12T03:20:17.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-12T03:20:17.638 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:20:17.939 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:17.939 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:17.939 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:17.939 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: Updating smithi007:/etc/ceph/ceph.conf 2024-03-12T03:20:17.939 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-12T03:20:17.939 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-12T03:20:17.939 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.939 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.939 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.939 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.940 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.940 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.940 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:17.940 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:20:17.940 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:20:17.940 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:17.940 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:17 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:18.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:18.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:18.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:18.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: Updating smithi007:/etc/ceph/ceph.conf 2024-03-12T03:20:18.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-12T03:20:18.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-12T03:20:18.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:18.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:18.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:18.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:18.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:18.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:18.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:18.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:20:18.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:20:18.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:18.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:17 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:18.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:18.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:18.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:18.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: Updating smithi007:/etc/ceph/ceph.conf 2024-03-12T03:20:18.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-12T03:20:18.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-12T03:20:18.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:18.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:18.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:18.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:18.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:18.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:18.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:18.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:20:18.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:20:18.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:18.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:17 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:20:19.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:18 smithi033 ceph-mon[27552]: Updating smithi007:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:20:19.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:18 smithi033 ceph-mon[27552]: Updating smithi033:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:20:19.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:18 smithi033 ceph-mon[27552]: Updating smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:20:19.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:18 smithi033 ceph-mon[27552]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-12T03:20:19.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:18 smithi033 ceph-mon[27552]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T03:20:19.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:18 smithi033 ceph-mon[27552]: Reconfiguring daemon mon.a on smithi007 2024-03-12T03:20:19.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:18 smithi033 ceph-mon[27552]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:19.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:18 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:19.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:18 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:19.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:18 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:19.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:18 smithi007 ceph-mon[25800]: Updating smithi007:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:20:19.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:18 smithi007 ceph-mon[25800]: Updating smithi033:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:20:19.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:18 smithi007 ceph-mon[25800]: Updating smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:20:19.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:18 smithi007 ceph-mon[25800]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-12T03:20:19.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:18 smithi007 ceph-mon[25800]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T03:20:19.029 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:18 smithi007 ceph-mon[25800]: Reconfiguring daemon mon.a on smithi007 2024-03-12T03:20:19.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:18 smithi007 ceph-mon[25800]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:19.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:18 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:19.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:18 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:19.030 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:18 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:19.043 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:18 smithi170 ceph-mon[27708]: Updating smithi007:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:20:19.043 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:18 smithi170 ceph-mon[27708]: Updating smithi033:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:20:19.043 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:18 smithi170 ceph-mon[27708]: Updating smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:20:19.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:18 smithi170 ceph-mon[27708]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-12T03:20:19.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:18 smithi170 ceph-mon[27708]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T03:20:19.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:18 smithi170 ceph-mon[27708]: Reconfiguring daemon mon.a on smithi007 2024-03-12T03:20:19.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:18 smithi170 ceph-mon[27708]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:19.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:18 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:19.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:18 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:19.044 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:18 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:19.750 INFO:teuthology.orchestra.run.smithi007.stdout:# minimal ceph.conf for fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:20:19.750 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-03-12T03:20:19.750 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:20:19.750 INFO:teuthology.orchestra.run.smithi007.stdout: mon_host = [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2024-03-12T03:20:19.894 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:19 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/1828430363' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:20.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:19 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/1828430363' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:20.108 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:19 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/1828430363' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:20.439 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-12T03:20:20.440 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-12T03:20:20.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T03:20:20.469 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-12T03:20:20.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:20:20.537 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-12T03:20:20.537 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T03:20:20.568 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-12T03:20:20.568 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:20:20.636 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-12T03:20:20.636 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T03:20:20.666 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-12T03:20:20.666 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:20:20.734 INFO:tasks.cephadm:Adding mgr.y on smithi007 2024-03-12T03:20:20.734 INFO:tasks.cephadm:Adding mgr.x on smithi033 2024-03-12T03:20:20.734 INFO:tasks.cephadm:Adding mgr.z on smithi170 2024-03-12T03:20:20.737 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph orch apply mgr '3;smithi007=y;smithi033=x;smithi170=z' 2024-03-12T03:20:21.005 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T03:20:21.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:21 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:21 smithi007 ceph-mon[25800]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:21.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:21 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:21 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:21 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:21 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:21 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:21 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:21 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:21 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:21 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:20:21.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:21 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:20:21.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:21 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:21.349 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:21 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.349 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:21 smithi170 ceph-mon[27708]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:21.349 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:21 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.349 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:21 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:21 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:21 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:21 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:21 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:21 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:21 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:21 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:20:21.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:21 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:20:21.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:21 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:21.410 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:21 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.410 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:21 smithi033 ceph-mon[27552]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:21.411 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:21 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.411 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:21 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.411 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:21 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.411 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:21 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.411 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:21 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.411 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:21 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.411 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:21 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.411 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:21 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:21.411 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:21 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:20:21.411 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:21 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:20:21.411 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:21 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:22.279 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:22 smithi170 ceph-mon[27708]: Reconfiguring mon.b (monmap changed)... 2024-03-12T03:20:22.280 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:22 smithi170 ceph-mon[27708]: Reconfiguring daemon mon.b on smithi033 2024-03-12T03:20:22.280 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:22 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:22.318 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:22 smithi033 ceph-mon[27552]: Reconfiguring mon.b (monmap changed)... 2024-03-12T03:20:22.318 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:22 smithi033 ceph-mon[27552]: Reconfiguring daemon mon.b on smithi033 2024-03-12T03:20:22.318 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:22 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:22.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:22 smithi007 ceph-mon[25800]: Reconfiguring mon.b (monmap changed)... 2024-03-12T03:20:22.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:22 smithi007 ceph-mon[25800]: Reconfiguring daemon mon.b on smithi033 2024-03-12T03:20:22.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:22 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:22.737 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled mgr update... 2024-03-12T03:20:23.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:23 smithi007 ceph-mon[25800]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:23.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:23 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:23.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:23 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:23.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:23 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:20:23.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:23 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:20:23.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:23 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:23.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:23 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:23.299 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:23 smithi170 ceph-mon[27708]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:23.299 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:23 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:23.299 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:23 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:23.299 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:23 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:20:23.299 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:23 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:20:23.300 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:23 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:23.300 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:23 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:23.301 DEBUG:teuthology.orchestra.run.smithi033:mgr.x> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mgr.x.service 2024-03-12T03:20:23.303 DEBUG:teuthology.orchestra.run.smithi170:mgr.z> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mgr.z.service 2024-03-12T03:20:23.305 INFO:tasks.cephadm:Deploying OSDs... 2024-03-12T03:20:23.305 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-12T03:20:23.305 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:20:23.325 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-12T03:20:23.325 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/nvme1n1 2024-03-12T03:20:23.330 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:23 smithi033 ceph-mon[27552]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:23.330 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:23 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:23.330 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:23 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:23.330 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:23 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:20:23.330 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:23 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:20:23.330 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:23 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:23.330 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:23 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:23.382 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/nvme1n1 2024-03-12T03:20:23.383 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T03:20:23.383 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-12T03:20:23.383 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T03:20:23.383 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T03:20:23.383 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-12 03:20:06.017797773 +0000 2024-03-12T03:20:23.383 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-12 03:16:57.676509770 +0000 2024-03-12T03:20:23.383 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-12 03:16:57.676509770 +0000 2024-03-12T03:20:23.383 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-12 03:16:57.669509878 +0000 2024-03-12T03:20:23.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-12T03:20:23.454 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-12T03:20:23.455 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-12T03:20:23.455 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000138317 s, 3.7 MB/s 2024-03-12T03:20:23.456 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-12T03:20:23.513 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/nvme2n1 2024-03-12T03:20:23.568 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/nvme2n1 2024-03-12T03:20:23.569 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T03:20:23.569 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-12T03:20:23.569 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T03:20:23.569 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T03:20:23.569 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-12 03:20:06.025797650 +0000 2024-03-12T03:20:23.569 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-12 03:16:57.821507541 +0000 2024-03-12T03:20:23.569 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-12 03:16:57.821507541 +0000 2024-03-12T03:20:23.569 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-12 03:16:57.812507679 +0000 2024-03-12T03:20:23.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-12T03:20:23.636 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-12T03:20:23.636 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-12T03:20:23.636 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000119282 s, 4.3 MB/s 2024-03-12T03:20:23.637 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-12T03:20:23.693 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/nvme3n1 2024-03-12T03:20:23.749 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/nvme3n1 2024-03-12T03:20:23.749 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T03:20:23.749 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-03-12T03:20:23.749 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T03:20:23.749 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T03:20:23.749 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-12 03:20:06.034797512 +0000 2024-03-12T03:20:23.749 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-12 03:16:57.958505435 +0000 2024-03-12T03:20:23.749 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-12 03:16:57.958505435 +0000 2024-03-12T03:20:23.750 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-12 03:16:57.955505481 +0000 2024-03-12T03:20:23.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-12T03:20:23.815 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-12T03:20:23.815 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-12T03:20:23.815 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000114072 s, 4.5 MB/s 2024-03-12T03:20:23.816 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-12T03:20:23.871 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/nvme4n1 2024-03-12T03:20:23.927 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/nvme4n1 2024-03-12T03:20:23.927 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T03:20:23.927 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 800 Links: 1 Device type: 103,8 2024-03-12T03:20:23.927 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T03:20:23.927 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T03:20:23.927 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-12 03:20:06.042797389 +0000 2024-03-12T03:20:23.927 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-12 03:16:58.110503098 +0000 2024-03-12T03:20:23.927 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-12 03:16:58.110503098 +0000 2024-03-12T03:20:23.928 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-12 03:16:58.105503175 +0000 2024-03-12T03:20:23.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-12T03:20:23.993 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-12T03:20:23.993 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-12T03:20:23.993 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000121836 s, 4.2 MB/s 2024-03-12T03:20:23.994 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-12T03:20:24.050 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-12T03:20:24.050 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:20:24.068 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-12T03:20:24.068 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/nvme1n1 2024-03-12T03:20:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:24 smithi007 ceph-mon[25800]: Reconfiguring mon.c (monmap changed)... 2024-03-12T03:20:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:24 smithi007 ceph-mon[25800]: Reconfiguring daemon mon.c on smithi170 2024-03-12T03:20:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:24 smithi007 ceph-mon[25800]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi007=y;smithi033=x;smithi170=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:24 smithi007 ceph-mon[25800]: Saving service mgr spec with placement smithi007=y;smithi033=x;smithi170=z;count:3 2024-03-12T03:20:24.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:24 smithi170 ceph-mon[27708]: Reconfiguring mon.c (monmap changed)... 2024-03-12T03:20:24.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:24 smithi170 ceph-mon[27708]: Reconfiguring daemon mon.c on smithi170 2024-03-12T03:20:24.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:24 smithi170 ceph-mon[27708]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi007=y;smithi033=x;smithi170=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:24.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:24 smithi170 ceph-mon[27708]: Saving service mgr spec with placement smithi007=y;smithi033=x;smithi170=z;count:3 2024-03-12T03:20:24.125 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/nvme1n1 2024-03-12T03:20:24.126 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T03:20:24.126 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-12T03:20:24.126 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T03:20:24.126 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T03:20:24.126 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-12 03:20:19.160240683 +0000 2024-03-12T03:20:24.126 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-12 03:16:59.288439752 +0000 2024-03-12T03:20:24.126 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-12 03:16:59.288439752 +0000 2024-03-12T03:20:24.126 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-12 03:16:59.277438137 +0000 2024-03-12T03:20:24.126 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-12T03:20:24.196 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-12T03:20:24.196 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-12T03:20:24.196 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000402102 s, 1.3 MB/s 2024-03-12T03:20:24.197 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-12T03:20:24.255 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/nvme2n1 2024-03-12T03:20:24.312 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/nvme2n1 2024-03-12T03:20:24.312 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T03:20:24.312 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-12T03:20:24.312 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T03:20:24.312 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T03:20:24.312 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-12 03:20:19.171240505 +0000 2024-03-12T03:20:24.312 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-12 03:16:59.448463248 +0000 2024-03-12T03:20:24.312 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-12 03:16:59.448463248 +0000 2024-03-12T03:20:24.312 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-12 03:16:59.435461339 +0000 2024-03-12T03:20:24.313 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-12T03:20:24.377 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:24 smithi033 ceph-mon[27552]: Reconfiguring mon.c (monmap changed)... 2024-03-12T03:20:24.377 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:24 smithi033 ceph-mon[27552]: Reconfiguring daemon mon.c on smithi170 2024-03-12T03:20:24.377 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:24 smithi033 ceph-mon[27552]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi007=y;smithi033=x;smithi170=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:24.378 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:24 smithi033 ceph-mon[27552]: Saving service mgr spec with placement smithi007=y;smithi033=x;smithi170=z;count:3 2024-03-12T03:20:24.380 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-12T03:20:24.380 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-12T03:20:24.380 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000226388 s, 2.3 MB/s 2024-03-12T03:20:24.381 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-12T03:20:24.438 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/nvme3n1 2024-03-12T03:20:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/nvme3n1 2024-03-12T03:20:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T03:20:24.496 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-03-12T03:20:24.496 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T03:20:24.496 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T03:20:24.496 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-12 03:20:19.182240327 +0000 2024-03-12T03:20:24.496 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-12 03:16:59.594484688 +0000 2024-03-12T03:20:24.496 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-12 03:16:59.594484688 +0000 2024-03-12T03:20:24.496 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-12 03:16:59.581482779 +0000 2024-03-12T03:20:24.496 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-12T03:20:24.563 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-12T03:20:24.564 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-12T03:20:24.564 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000312821 s, 1.6 MB/s 2024-03-12T03:20:24.565 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-12T03:20:24.621 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/nvme4n1 2024-03-12T03:20:24.677 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/nvme4n1 2024-03-12T03:20:24.677 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T03:20:24.677 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-12T03:20:24.677 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T03:20:24.677 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T03:20:24.677 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-12 03:20:19.193240148 +0000 2024-03-12T03:20:24.678 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-12 03:16:59.740489722 +0000 2024-03-12T03:20:24.678 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-12 03:16:59.740489722 +0000 2024-03-12T03:20:24.678 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-12 03:16:59.729489902 +0000 2024-03-12T03:20:24.678 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-12T03:20:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-12T03:20:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-12T03:20:24.743 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000192308 s, 2.7 MB/s 2024-03-12T03:20:24.744 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-12T03:20:24.800 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-12T03:20:24.800 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:20:24.815 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-12T03:20:24.815 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/nvme1n1 2024-03-12T03:20:24.870 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/nvme1n1 2024-03-12T03:20:24.871 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T03:20:24.871 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-12T03:20:24.871 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T03:20:24.871 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T03:20:24.871 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-12 03:20:19.729250715 +0000 2024-03-12T03:20:24.871 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-12 03:17:00.962094638 +0000 2024-03-12T03:20:24.871 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-12 03:17:00.962094638 +0000 2024-03-12T03:20:24.871 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-12 03:17:00.952094882 +0000 2024-03-12T03:20:24.871 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-12T03:20:24.938 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-12T03:20:24.938 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-12T03:20:24.938 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000134129 s, 3.8 MB/s 2024-03-12T03:20:24.939 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-12T03:20:24.995 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/nvme2n1 2024-03-12T03:20:25.051 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/nvme2n1 2024-03-12T03:20:25.051 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T03:20:25.051 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-12T03:20:25.051 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T03:20:25.051 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T03:20:25.051 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-12 03:20:19.738250496 +0000 2024-03-12T03:20:25.051 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-12 03:17:01.109091056 +0000 2024-03-12T03:20:25.051 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-12 03:17:01.109091056 +0000 2024-03-12T03:20:25.051 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-12 03:17:01.100091276 +0000 2024-03-12T03:20:25.052 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-12T03:20:25.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:25 smithi007 ceph-mon[25800]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:25.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:25.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:25.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:25.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:25.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:25.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:25.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:20:25.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:20:25.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:20:25.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:25 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:25.116 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-12T03:20:25.116 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-12T03:20:25.116 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000103183 s, 5.0 MB/s 2024-03-12T03:20:25.118 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-12T03:20:25.173 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/nvme3n1 2024-03-12T03:20:25.227 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/nvme3n1 2024-03-12T03:20:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T03:20:25.228 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-12T03:20:25.228 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T03:20:25.228 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T03:20:25.228 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-12 03:20:19.746250301 +0000 2024-03-12T03:20:25.228 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-12 03:17:01.250087620 +0000 2024-03-12T03:20:25.228 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-12 03:17:01.250087620 +0000 2024-03-12T03:20:25.228 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-12 03:17:01.246087718 +0000 2024-03-12T03:20:25.228 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-12T03:20:25.292 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:25 smithi170 ceph-mon[27708]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:25.292 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:25 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:25.292 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:25 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:25.292 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:25 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:25.292 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:25 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:25.292 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:25 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:25.292 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:25 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:25.292 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:25 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:20:25.292 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:25 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:20:25.292 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:25 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:20:25.292 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:25 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:25.294 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-12T03:20:25.294 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-12T03:20:25.294 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000101485 s, 5.0 MB/s 2024-03-12T03:20:25.295 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-12T03:20:25.352 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/nvme4n1 2024-03-12T03:20:25.406 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/nvme4n1 2024-03-12T03:20:25.406 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-12T03:20:25.407 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-12T03:20:25.407 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-12T03:20:25.407 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-12T03:20:25.407 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-12 03:20:19.755250082 +0000 2024-03-12T03:20:25.407 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-12 03:17:01.410083722 +0000 2024-03-12T03:20:25.407 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-12 03:17:01.410083722 +0000 2024-03-12T03:20:25.407 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-12 03:17:01.403083892 +0000 2024-03-12T03:20:25.407 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-12T03:20:25.475 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-12T03:20:25.475 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-12T03:20:25.475 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000107327 s, 4.8 MB/s 2024-03-12T03:20:25.476 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-12T03:20:25.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:25 smithi033 ceph-mon[27552]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:25.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:25 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:25.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:25 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:25.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:25 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:25.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:25 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:25.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:25 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:25.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:25 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:25.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:25 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:20:25.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:25 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:20:25.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:25 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:20:25.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:25 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:25.532 INFO:tasks.cephadm:Deploying osd.0 on smithi007 with /dev/nvme4n1... 2024-03-12T03:20:25.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-12T03:20:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:20:26.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:26 smithi007 ceph-mon[25800]: Deploying daemon mgr.z on smithi170 2024-03-12T03:20:26.274 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:26 smithi170 ceph-mon[27708]: Deploying daemon mgr.z on smithi170 2024-03-12T03:20:26.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:26 smithi033 ceph-mon[27552]: Deploying daemon mgr.z on smithi170 2024-03-12T03:20:26.958 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:26 smithi170 ceph-mgr[30239]: -- 172.21.15.170:0/2307806191 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56398c3305a0 con 0x56398c304c00 2024-03-12T03:20:26.959 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:26 smithi170 bash[30201]: fe9479a3eaa95670a9da91d153de6b7f217833020408f423f7a78c2036c5b7f5 2024-03-12T03:20:27.310 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:27 smithi007 ceph-mon[25800]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:27.310 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:27 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:27.310 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:27 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:27.363 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:26 smithi170 systemd[1]: Started Ceph mgr.z for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:20:27.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:27 smithi170 ceph-mon[27708]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:27.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:27 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:27.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:27 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:27.414 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:27 smithi033 ceph-mon[27552]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:27.414 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:27 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:27.414 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:27 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:27.668 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/nvme4n1 2024-03-12T03:20:27.668 INFO:teuthology.orchestra.run.smithi007.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-12T03:20:27.668 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-12T03:20:27.668 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-03-12T03:20:27.668 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-03-12T03:20:27.668 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00953004 s, 1.1 GB/s 2024-03-12T03:20:27.668 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-03-12T03:20:27.863 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:27 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:27.406+0000 7fda369e2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:20:27.863 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:27 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:27.498+0000 7fda369e2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:20:28.188 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:28 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:28.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:28 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:20:28.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:28 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:20:28.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:28 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:20:28.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:28 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:28.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:28 smithi007 ceph-mon[25800]: Deploying daemon mgr.x on smithi033 2024-03-12T03:20:28.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:28 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:28.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:28 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:28.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:28 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:28.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi007:/dev/nvme4n1 2024-03-12T03:20:28.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:28.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:20:28.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:20:28.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:20:28.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:28.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-mon[27708]: Deploying daemon mgr.x on smithi033 2024-03-12T03:20:28.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:28.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:28.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:28.364 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:28.135+0000 7fda369e2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:20:28.364 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:28.227+0000 7fda369e2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:20:28.390 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:28 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:28.390 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:28 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:20:28.390 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:28 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:20:28.390 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:28 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:20:28.390 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:28 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:28.391 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:28 smithi033 ceph-mon[27552]: Deploying daemon mgr.x on smithi033 2024-03-12T03:20:28.391 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:28 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:28.391 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:28 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:28.391 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:28 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:28.456 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:20:28.821 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:28.376+0000 7fda369e2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:20:28.821 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:28.469+0000 7fda369e2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:20:28.822 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:28.550+0000 7fda369e2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:20:28.822 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:28.819+0000 7fda369e2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:20:29.104 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:29 smithi170 ceph-mon[27708]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:29.104 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:28 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:28.926+0000 7fda369e2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:20:29.105 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:29 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:29.009+0000 7fda369e2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:20:29.105 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:29 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:29.102+0000 7fda369e2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:20:29.160 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:29 smithi007 ceph-mon[25800]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:29.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:29 smithi033 ceph-mon[27552]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:29.263 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:29 smithi033 systemd[1]: Starting Ceph mgr.x for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:20:29.363 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:29 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:29.277+0000 7fda369e2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:20:29.651 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:29 smithi033 podman[29222]: 2024-03-12 03:20:29.291427689 +0000 UTC m=+0.017326784 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:20:29.651 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:29 smithi033 podman[29222]: 2024-03-12 03:20:29.391076597 +0000 UTC m=+0.116975673 container create ec8321b1b7ab2e05a673b2ccde97114ee39cb927cc7c669e2b3a334ff2c9700f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-12T03:20:29.651 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:29 smithi033 podman[29222]: 2024-03-12 03:20:29.618348379 +0000 UTC m=+0.344247456 container init ec8321b1b7ab2e05a673b2ccde97114ee39cb927cc7c669e2b3a334ff2c9700f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-12T03:20:29.651 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:29 smithi033 podman[29222]: 2024-03-12 03:20:29.623799218 +0000 UTC m=+0.349698295 container start ec8321b1b7ab2e05a673b2ccde97114ee39cb927cc7c669e2b3a334ff2c9700f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:20:29.651 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:29 smithi033 ceph-mgr[29260]: -- 172.21.15.33:0/1341537598 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557042abe5a0 con 0x557042a92c00 2024-03-12T03:20:30.013 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:29 smithi033 bash[29222]: ec8321b1b7ab2e05a673b2ccde97114ee39cb927cc7c669e2b3a334ff2c9700f 2024-03-12T03:20:30.013 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:29 smithi033 systemd[1]: Started Ceph mgr.x for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:20:30.513 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:30.191+0000 7f3a8fd48200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:20:30.513 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:30.287+0000 7f3a8fd48200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:20:30.609 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:30.238+0000 7fda369e2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:20:30.609 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:30.365+0000 7fda369e2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:20:30.863 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:30.712+0000 7fda369e2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:20:30.863 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:30.820+0000 7fda369e2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:20:30.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:30.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:30.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:30.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:30.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:30.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:30.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:30.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:30.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-12T03:20:30.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:30.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:20:30.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:20:30.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:20:30.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:30.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:20:30.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:30 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:31.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:31.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:31.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:31.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:31.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:31.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:31.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:31.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:31.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-12T03:20:31.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:31.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:20:31.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:20:31.005 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:20:31.006 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:31.006 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:20:31.006 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:30 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:31.012 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:30.953+0000 7f3a8fd48200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:20:31.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:31.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:31.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:31.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:31.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:31.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:31.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:31.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:31.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-12T03:20:31.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:31.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:20:31.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:20:31.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:20:31.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:31.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:20:31.014 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:30 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:31.205 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:31 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:31.021+0000 7fda369e2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:20:31.205 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:31 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:31.201+0000 7fda369e2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:20:31.379 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:31 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:31.049+0000 7f3a8fd48200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:20:31.379 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:31 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:31.193+0000 7f3a8fd48200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:20:31.379 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:31 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:31.292+0000 7f3a8fd48200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:20:31.379 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:31 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:31.376+0000 7f3a8fd48200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:20:31.790 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:31 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:31.500+0000 7fda369e2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:20:31.944 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:31 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:31.644+0000 7f3a8fd48200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:20:31.944 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:31 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:31.757+0000 7f3a8fd48200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:20:31.944 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:31 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:31.845+0000 7f3a8fd48200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:20:31.944 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:31 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:31.941+0000 7f3a8fd48200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:20:32.109 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:31 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:31.789+0000 7fda369e2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:20:32.109 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:32 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:32.036+0000 7fda369e2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:20:32.109 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:31 smithi170 ceph-mon[27708]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:32.109 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:31 smithi170 ceph-mon[27708]: Reconfiguring mgr.y (unknown last config time)... 2024-03-12T03:20:32.109 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:31 smithi170 ceph-mon[27708]: Reconfiguring daemon mgr.y on smithi007 2024-03-12T03:20:32.109 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:31 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:32.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:31 smithi033 ceph-mon[27552]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:32.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:31 smithi033 ceph-mon[27552]: Reconfiguring mgr.y (unknown last config time)... 2024-03-12T03:20:32.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:31 smithi033 ceph-mon[27552]: Reconfiguring daemon mgr.y on smithi007 2024-03-12T03:20:32.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:31 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:32.263 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:32 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:32.120+0000 7f3a8fd48200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:20:32.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:31 smithi007 ceph-mon[25800]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:32.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:31 smithi007 ceph-mon[25800]: Reconfiguring mgr.y (unknown last config time)... 2024-03-12T03:20:32.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:31 smithi007 ceph-mon[25800]: Reconfiguring daemon mgr.y on smithi007 2024-03-12T03:20:32.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:31 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:33.263 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:33 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:33.129+0000 7f3a8fd48200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:20:33.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:33 smithi033 ceph-mon[27552]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:33.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:33 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:33.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:33 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:33.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:33 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:33.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:33 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:33.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:33 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:33.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:33 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:33.289 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:32 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:32.909+0000 7fda369e2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:20:33.289 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:33 smithi170 ceph-mon[27708]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:33.289 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:33 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:33.290 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:33 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:33.290 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:33 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:33.290 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:33 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:33.290 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:33 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:33.290 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:33 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:33.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:33 smithi007 ceph-mon[25800]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:33.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:33 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:33.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:33 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:33.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:33 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:33.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:33 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:33.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:33 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:33.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:33 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:33.538 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:33 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:33.271+0000 7f3a8fd48200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:20:33.610 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:33 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:33.510+0000 7fda369e2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:20:33.610 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:33 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:33.606+0000 7fda369e2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:20:33.842 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:33 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:33.618+0000 7f3a8fd48200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:20:33.843 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:33 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:33.711+0000 7f3a8fd48200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:20:33.863 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:20:33 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:20:33.841+0000 7fda369e2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:20:34.117 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:33 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:33.925+0000 7f3a8fd48200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:20:34.118 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:34.114+0000 7f3a8fd48200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:20:34.118 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-mon[27552]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-12T03:20:34.118 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:34.118 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:34.118 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:34.118 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:34.118 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-mon[27552]: Standby manager daemon z started 2024-03-12T03:20:34.118 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.170:0/985524991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T03:20:34.118 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.170:0/985524991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:20:34.118 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.170:0/985524991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T03:20:34.119 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.170:0/985524991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:20:34.119 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/2767055128' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9444b255-7dd8-476c-a981-596313fa6856"}]: dispatch 2024-03-12T03:20:34.119 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/2767055128' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9444b255-7dd8-476c-a981-596313fa6856"}]': finished 2024-03-12T03:20:34.119 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-mon[27552]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T03:20:34.119 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:34.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:34 smithi007 ceph-mon[25800]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-12T03:20:34.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:34 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:34.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:34 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:34.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:34 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:34.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:34 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:34.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:34 smithi007 ceph-mon[25800]: Standby manager daemon z started 2024-03-12T03:20:34.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:34 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.170:0/985524991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T03:20:34.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:34 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.170:0/985524991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:20:34.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:34 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.170:0/985524991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T03:20:34.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:34 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.170:0/985524991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:20:34.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:34 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/2767055128' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9444b255-7dd8-476c-a981-596313fa6856"}]: dispatch 2024-03-12T03:20:34.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:34 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/2767055128' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9444b255-7dd8-476c-a981-596313fa6856"}]': finished 2024-03-12T03:20:34.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:34 smithi007 ceph-mon[25800]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T03:20:34.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:34 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:34.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:34 smithi170 ceph-mon[27708]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-12T03:20:34.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:34 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:34.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:34 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:34.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:34 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:34.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:34 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:34.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:34 smithi170 ceph-mon[27708]: Standby manager daemon z started 2024-03-12T03:20:34.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:34 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.170:0/985524991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T03:20:34.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:34 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.170:0/985524991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:20:34.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:34 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.170:0/985524991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T03:20:34.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:34 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.170:0/985524991' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:20:34.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:34 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/2767055128' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9444b255-7dd8-476c-a981-596313fa6856"}]: dispatch 2024-03-12T03:20:34.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:34 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/2767055128' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9444b255-7dd8-476c-a981-596313fa6856"}]': finished 2024-03-12T03:20:34.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:34 smithi170 ceph-mon[27708]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T03:20:34.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:34 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:34.707 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:34.425+0000 7f3a8fd48200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:20:34.707 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:34.703+0000 7f3a8fd48200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:20:35.263 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:34 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:34.959+0000 7f3a8fd48200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:20:35.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:35 smithi033 ceph-mon[27552]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:35.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:35 smithi033 ceph-mon[27552]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-12T03:20:35.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:35 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T03:20:35.325 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:35 smithi007 ceph-mon[25800]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:35.325 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:35 smithi007 ceph-mon[25800]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-12T03:20:35.325 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:35 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T03:20:35.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:35 smithi170 ceph-mon[27708]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:35.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:35 smithi170 ceph-mon[27708]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-12T03:20:35.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:35 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T03:20:36.128 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:35 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:35.849+0000 7f3a8fd48200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:20:36.128 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:36 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/3747759724' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:20:36.128 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:36 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:36.129 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:36 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:36.129 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:36 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:36.129 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:36 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:36.129 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:36 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:36.129 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:36 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:36.129 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:36 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:36.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:36 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/3747759724' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:20:36.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:36 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:36.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:36 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:36.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:36 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:36.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:36 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:36.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:36 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:36.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:36 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:36.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:36 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:36.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:36 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/3747759724' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:20:36.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:36 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:36.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:36 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:36.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:36 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:36.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:36 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:36.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:36 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:36.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:36 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:36.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:36 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:36.762 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:36 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:36.441+0000 7f3a8fd48200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:20:36.763 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:36 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:36.542+0000 7f3a8fd48200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:20:37.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:20:36 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:20:36.761+0000 7f3a8fd48200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:20:37.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:37 smithi007 ceph-mon[25800]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:37.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:37 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:37.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:37 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:37.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:37 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:37.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:37 smithi007 ceph-mon[25800]: Standby manager daemon x started 2024-03-12T03:20:37.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:37 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.33:0/2947884145' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:20:37.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:37 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.33:0/2947884145' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:20:37.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:37 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.33:0/2947884145' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:20:37.309 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:37 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.33:0/2947884145' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:20:37.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:37 smithi170 ceph-mon[27708]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:37.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:37 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:37.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:37 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:37.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:37 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:37.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:37 smithi170 ceph-mon[27708]: Standby manager daemon x started 2024-03-12T03:20:37.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:37 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.33:0/2947884145' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:20:37.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:37 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.33:0/2947884145' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:20:37.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:37 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.33:0/2947884145' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:20:37.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:37 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.33:0/2947884145' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:20:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:37 smithi033 ceph-mon[27552]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:37 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:37 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:37 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:37 smithi033 ceph-mon[27552]: Standby manager daemon x started 2024-03-12T03:20:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:37 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.33:0/2947884145' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:20:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:37 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.33:0/2947884145' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:20:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:37 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.33:0/2947884145' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:20:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:37 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.33:0/2947884145' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:20:38.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:38 smithi007 ceph-mon[25800]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-12T03:20:38.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:38 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:20:38.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:38 smithi007 ceph-mon[25800]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:38.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:38 smithi170 ceph-mon[27708]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-12T03:20:38.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:38 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:20:38.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:38 smithi170 ceph-mon[27708]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:38.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:38 smithi033 ceph-mon[27552]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-12T03:20:38.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:38 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:20:38.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:38 smithi033 ceph-mon[27552]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:40.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:40 smithi033 ceph-mon[27552]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:40.574 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:40 smithi007 ceph-mon[25800]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:40.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:40 smithi170 ceph-mon[27708]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:41.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:41 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T03:20:41.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:41 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:41.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:41 smithi033 ceph-mon[27552]: Deploying daemon osd.0 on smithi007 2024-03-12T03:20:41.531 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:41 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T03:20:41.531 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:41 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:41.531 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:41 smithi007 ceph-mon[25800]: Deploying daemon osd.0 on smithi007 2024-03-12T03:20:41.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:41 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T03:20:41.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:41 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:41.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:41 smithi170 ceph-mon[27708]: Deploying daemon osd.0 on smithi007 2024-03-12T03:20:42.495 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:42 smithi007 ceph-mon[25800]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:42.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:42 smithi033 ceph-mon[27552]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:42.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:42 smithi170 ceph-mon[27708]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:44.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:44 smithi033 ceph-mon[27552]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:44.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:44 smithi007 ceph-mon[25800]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:44.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:44 smithi170 ceph-mon[27708]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:46.502 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:46 smithi007 ceph-mon[25800]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:46.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:46 smithi033 ceph-mon[27552]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:46.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:46 smithi170 ceph-mon[27708]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:48.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:48 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:48 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:48.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:48 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:48 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:48.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:48 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:48.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:48 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:48 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:48 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:48 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:48 smithi007 ceph-mon[25800]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:48 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:48 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:48 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:48 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:48 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:48 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:48 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:48 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:48 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.764 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:48 smithi033 ceph-mon[27552]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:48.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:48 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:48 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:48 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:48 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:48 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:48 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:48 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:48 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:48 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:48 smithi170 ceph-mon[27708]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:50.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:50 smithi033 ceph-mon[27552]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:50.592 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:50 smithi007 ceph-mon[25800]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:50.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:50 smithi170 ceph-mon[27708]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:51.424 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:51 smithi007 ceph-mon[25800]: from='osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:20:51.495 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 0 on host 'smithi007' 2024-03-12T03:20:51.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:51 smithi033 ceph-mon[27552]: from='osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:20:51.518 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:51 smithi170 ceph-mon[27708]: from='osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:20:52.121 DEBUG:teuthology.orchestra.run.smithi007:osd.0> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@osd.0.service 2024-03-12T03:20:52.124 INFO:tasks.cephadm:Deploying osd.1 on smithi033 with /dev/nvme4n1... 2024-03-12T03:20:52.124 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-12T03:20:52.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-mon[25800]: from='osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:20:52.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-mon[25800]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T03:20:52.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-mon[25800]: from='osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-12T03:20:52.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:52.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:52.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:52.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:52.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:52.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-mon[25800]: Adjusting osd_memory_target on smithi007 to 16069M 2024-03-12T03:20:52.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:52.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:52.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:52.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:52.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-mon[25800]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:52.379 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:20:52.499 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:52 smithi170 ceph-mon[27708]: from='osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:20:52.499 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:52 smithi170 ceph-mon[27708]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T03:20:52.500 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:52 smithi170 ceph-mon[27708]: from='osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-12T03:20:52.500 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:52.500 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:52.500 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:52.500 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:52.500 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:52.500 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:52 smithi170 ceph-mon[27708]: Adjusting osd_memory_target on smithi007 to 16069M 2024-03-12T03:20:52.500 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:52.500 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:52.500 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:52.500 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:52.500 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:52 smithi170 ceph-mon[27708]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:52.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:52 smithi033 ceph-mon[27552]: from='osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:20:52.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:52 smithi033 ceph-mon[27552]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T03:20:52.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:52 smithi033 ceph-mon[27552]: from='osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-12T03:20:52.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:52.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:52.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:52.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:52.501 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:52.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:52 smithi033 ceph-mon[27552]: Adjusting osd_memory_target on smithi007 to 16069M 2024-03-12T03:20:52.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:52.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:52.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:52.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:52.502 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:52 smithi033 ceph-mon[27552]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:52.848 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-0[38020]: 2024-03-12T03:20:52.535+0000 7f8bad673700 -1 osd.0 0 waiting for initial osdmap 2024-03-12T03:20:52.848 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:20:52 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-0[38020]: 2024-03-12T03:20:52.538+0000 7f8ba545e700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:20:53.467 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:53 smithi170 ceph-mon[27708]: from='osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-03-12T03:20:53.468 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:53 smithi170 ceph-mon[27708]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T03:20:53.468 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:53.468 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:53.468 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:53 smithi170 ceph-mon[27708]: from='osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070]' entity='osd.0' 2024-03-12T03:20:53.468 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:53.468 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:53.468 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:53.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:53 smithi033 ceph-mon[27552]: from='osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-03-12T03:20:53.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:53 smithi033 ceph-mon[27552]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T03:20:53.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:53.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:53.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:53 smithi033 ceph-mon[27552]: from='osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070]' entity='osd.0' 2024-03-12T03:20:53.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:53.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:53.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:53.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:53 smithi007 ceph-mon[25800]: from='osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-03-12T03:20:53.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:53 smithi007 ceph-mon[25800]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T03:20:53.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:53.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:53.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:53 smithi007 ceph-mon[25800]: from='osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070]' entity='osd.0' 2024-03-12T03:20:53.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:53.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:53.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:54.391 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/nvme4n1 2024-03-12T03:20:54.392 INFO:teuthology.orchestra.run.smithi033.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-12T03:20:54.392 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-12T03:20:54.392 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-03-12T03:20:54.392 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-03-12T03:20:54.392 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140154 s, 748 MB/s 2024-03-12T03:20:54.392 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-03-12T03:20:54.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:54 smithi033 ceph-mon[27552]: purged_snaps scrub starts 2024-03-12T03:20:54.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:54 smithi033 ceph-mon[27552]: purged_snaps scrub ok 2024-03-12T03:20:54.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:54 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:54.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:54 smithi033 ceph-mon[27552]: osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070] boot 2024-03-12T03:20:54.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:54 smithi033 ceph-mon[27552]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T03:20:54.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:54 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:54.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:54 smithi033 ceph-mon[27552]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:54.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:54 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:54.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:54 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:54.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:54 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:54.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:54 smithi007 ceph-mon[25800]: purged_snaps scrub starts 2024-03-12T03:20:54.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:54 smithi007 ceph-mon[25800]: purged_snaps scrub ok 2024-03-12T03:20:54.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:54 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:54.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:54 smithi007 ceph-mon[25800]: osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070] boot 2024-03-12T03:20:54.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:54 smithi007 ceph-mon[25800]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T03:20:54.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:54 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:54.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:54 smithi007 ceph-mon[25800]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:54.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:54 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:54.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:54 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:54.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:54 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:54.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:54 smithi170 ceph-mon[27708]: purged_snaps scrub starts 2024-03-12T03:20:54.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:54 smithi170 ceph-mon[27708]: purged_snaps scrub ok 2024-03-12T03:20:54.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:54 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:54.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:54 smithi170 ceph-mon[27708]: osd.0 [v2:172.21.15.7:6802/4033945070,v1:172.21.15.7:6803/4033945070] boot 2024-03-12T03:20:54.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:54 smithi170 ceph-mon[27708]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T03:20:54.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:54 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:20:54.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:54 smithi170 ceph-mon[27708]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:20:54.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:54 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:54.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:54 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:54.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:54 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:20:55.425 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi033:/dev/nvme4n1 2024-03-12T03:20:55.853 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:20:56.704 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:56 smithi033 ceph-mon[27552]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T03:20:56.704 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:56 smithi033 ceph-mon[27552]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:20:56.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:56 smithi007 ceph-mon[25800]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T03:20:56.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:56 smithi007 ceph-mon[25800]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:20:56.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:56 smithi170 ceph-mon[27708]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T03:20:56.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:56 smithi170 ceph-mon[27708]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:20:58.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:58 smithi170 ceph-mon[27708]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:20:58.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:58 smithi033 ceph-mon[27552]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:20:58.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:58 smithi007 ceph-mon[25800]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:20:59.436 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:59 smithi033 ceph-mon[27552]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:59.436 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:59 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:20:59.436 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:59 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:20:59.436 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:20:59 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:59.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:59 smithi007 ceph-mon[25800]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:59.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:59 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:20:59.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:59 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:20:59.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:20:59 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:59.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:59 smithi170 ceph-mon[27708]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:59.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:59 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:20:59.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:59 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:20:59.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:20:59 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:00.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:00 smithi033 ceph-mon[27552]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:00.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:00 smithi007 ceph-mon[25800]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:00.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:00 smithi170 ceph-mon[27708]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:01.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:01 smithi033 ceph-mon[27552]: from='client.? 172.21.15.33:0/2988901493' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06e71f2a-abc2-43f6-af2c-67eb8cba66f8"}]: dispatch 2024-03-12T03:21:01.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:01 smithi033 ceph-mon[27552]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06e71f2a-abc2-43f6-af2c-67eb8cba66f8"}]: dispatch 2024-03-12T03:21:01.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:01 smithi033 ceph-mon[27552]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06e71f2a-abc2-43f6-af2c-67eb8cba66f8"}]': finished 2024-03-12T03:21:01.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:01 smithi033 ceph-mon[27552]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T03:21:01.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:01 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:01.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:01 smithi007 ceph-mon[25800]: from='client.? 172.21.15.33:0/2988901493' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06e71f2a-abc2-43f6-af2c-67eb8cba66f8"}]: dispatch 2024-03-12T03:21:01.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:01 smithi007 ceph-mon[25800]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06e71f2a-abc2-43f6-af2c-67eb8cba66f8"}]: dispatch 2024-03-12T03:21:01.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:01 smithi007 ceph-mon[25800]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06e71f2a-abc2-43f6-af2c-67eb8cba66f8"}]': finished 2024-03-12T03:21:01.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:01 smithi007 ceph-mon[25800]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T03:21:01.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:01 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:01.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:01 smithi170 ceph-mon[27708]: from='client.? 172.21.15.33:0/2988901493' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06e71f2a-abc2-43f6-af2c-67eb8cba66f8"}]: dispatch 2024-03-12T03:21:01.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:01 smithi170 ceph-mon[27708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06e71f2a-abc2-43f6-af2c-67eb8cba66f8"}]: dispatch 2024-03-12T03:21:01.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:01 smithi170 ceph-mon[27708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06e71f2a-abc2-43f6-af2c-67eb8cba66f8"}]': finished 2024-03-12T03:21:01.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:01 smithi170 ceph-mon[27708]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T03:21:01.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:01 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:02.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:02 smithi033 ceph-mon[27552]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:02.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:02 smithi033 ceph-mon[27552]: from='client.? 172.21.15.33:0/509318934' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:21:02.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:02 smithi007 ceph-mon[25800]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:02.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:02 smithi007 ceph-mon[25800]: from='client.? 172.21.15.33:0/509318934' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:21:02.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:02 smithi170 ceph-mon[27708]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:02.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:02 smithi170 ceph-mon[27708]: from='client.? 172.21.15.33:0/509318934' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:21:04.497 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:04 smithi033 ceph-mon[27552]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:04.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:04 smithi007 ceph-mon[25800]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:04.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:04 smithi170 ceph-mon[27708]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:06.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:06 smithi033 ceph-mon[27552]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:06.560 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:06 smithi007 ceph-mon[25800]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:06.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:06 smithi170 ceph-mon[27708]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:08.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:08 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:08.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:08 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:08.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:08 smithi170 ceph-mon[27708]: Detected new or changed devices on smithi007 2024-03-12T03:21:08.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:08 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:08.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:08 smithi170 ceph-mon[27708]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:08.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:08 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T03:21:08.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:08 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:08.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:08 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:08.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:08 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:08.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:08 smithi033 ceph-mon[27552]: Detected new or changed devices on smithi007 2024-03-12T03:21:08.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:08 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:08.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:08 smithi033 ceph-mon[27552]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:08.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:08 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T03:21:08.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:08 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:08.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:08 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:08.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:08 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:08.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:08 smithi007 ceph-mon[25800]: Detected new or changed devices on smithi007 2024-03-12T03:21:08.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:08 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:08.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:08 smithi007 ceph-mon[25800]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:08.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:08 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T03:21:08.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:08 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:09.500 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:09 smithi033 ceph-mon[27552]: Deploying daemon osd.1 on smithi033 2024-03-12T03:21:09.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:09 smithi007 ceph-mon[25800]: Deploying daemon osd.1 on smithi033 2024-03-12T03:21:09.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:09 smithi170 ceph-mon[27708]: Deploying daemon osd.1 on smithi033 2024-03-12T03:21:10.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:10 smithi033 ceph-mon[27552]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:10.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:10 smithi007 ceph-mon[25800]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:10.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:10 smithi170 ceph-mon[27708]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:12.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:12 smithi033 ceph-mon[27552]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:12.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:12 smithi170 ceph-mon[27708]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:12.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:12 smithi007 ceph-mon[25800]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:14.263 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.263 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.263 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.263 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.263 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.263 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:14.263 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:14.263 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:14.263 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.304 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.304 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.304 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.304 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.304 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.304 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:14.304 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:14.304 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:14.304 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:14.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:14.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:14.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:14.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:15.306 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:15 smithi033 ceph-mon[27552]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:15.307 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:15 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:15.307 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:15 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:15.307 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:15 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:15.307 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:15 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:15.307 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:15 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:15.307 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:15 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:15.307 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:15 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:15.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:15 smithi007 ceph-mon[25800]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:15.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:15 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:15.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:15 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:15.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:15 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:15.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:15 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:15.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:15 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:15.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:15 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:15.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:15 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:15.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:15 smithi170 ceph-mon[27708]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:15.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:15 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:15.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:15 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:15.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:15 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:15.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:15 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:15.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:15 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:15.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:15 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:15.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:15 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:16.557 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:16.557 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:16.557 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:16.558 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:16.558 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:16.558 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:16.558 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:16 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:16.558 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:16 smithi033 ceph-mon[27552]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:16.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:16.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:16.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:16.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:16.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:16.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:16.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:16 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:16.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:16 smithi007 ceph-mon[25800]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:16.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:16.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:16.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:16.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:16.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:16.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:16.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:16 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:16.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:16 smithi170 ceph-mon[27708]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:17.762 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:17 smithi033 ceph-mon[27552]: from='osd.1 [v2:172.21.15.33:6800/3621746125,v1:172.21.15.33:6801/3621746125]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:21:17.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:17 smithi033 ceph-mon[27552]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:21:17.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:17 smithi007 ceph-mon[25800]: from='osd.1 [v2:172.21.15.33:6800/3621746125,v1:172.21.15.33:6801/3621746125]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:21:17.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:17 smithi007 ceph-mon[25800]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:21:17.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:17 smithi170 ceph-mon[27708]: from='osd.1 [v2:172.21.15.33:6800/3621746125,v1:172.21.15.33:6801/3621746125]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:21:17.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:17 smithi170 ceph-mon[27708]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:21:18.594 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 1 on host 'smithi033' 2024-03-12T03:21:18.762 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:18 smithi033 ceph-mon[27552]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T03:21:18.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:18 smithi033 ceph-mon[27552]: from='osd.1 [v2:172.21.15.33:6800/3621746125,v1:172.21.15.33:6801/3621746125]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-12T03:21:18.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:18 smithi033 ceph-mon[27552]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T03:21:18.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:18 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:18.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:18 smithi033 ceph-mon[27552]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-12T03:21:18.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:18 smithi033 ceph-mon[27552]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:18 smithi007 ceph-mon[25800]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T03:21:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:18 smithi007 ceph-mon[25800]: from='osd.1 [v2:172.21.15.33:6800/3621746125,v1:172.21.15.33:6801/3621746125]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-12T03:21:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:18 smithi007 ceph-mon[25800]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T03:21:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:18 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:18 smithi007 ceph-mon[25800]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-12T03:21:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:18 smithi007 ceph-mon[25800]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:18.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:18 smithi170 ceph-mon[27708]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T03:21:18.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:18 smithi170 ceph-mon[27708]: from='osd.1 [v2:172.21.15.33:6800/3621746125,v1:172.21.15.33:6801/3621746125]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-12T03:21:18.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:18 smithi170 ceph-mon[27708]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T03:21:18.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:18 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:18.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:18 smithi170 ceph-mon[27708]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-12T03:21:18.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:18 smithi170 ceph-mon[27708]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:19.177 DEBUG:teuthology.orchestra.run.smithi033:osd.1> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@osd.1.service 2024-03-12T03:21:19.181 INFO:tasks.cephadm:Deploying osd.2 on smithi170 with /dev/nvme4n1... 2024-03-12T03:21:19.181 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-12T03:21:19.430 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T03:21:19.436 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:19 smithi033 ceph-mon[27552]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-12T03:21:19.436 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:19 smithi033 ceph-mon[27552]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T03:21:19.436 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:19 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:19.436 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:19 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:19.436 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:19 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:19.436 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:19 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:21:19.436 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:19 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:19.436 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:19 smithi033 ceph-mon[27552]: Adjusting osd_memory_target on smithi033 to 16069M 2024-03-12T03:21:19.436 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:19 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:19.436 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:19 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:19.437 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:19 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:19.437 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:19 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:19.437 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:19 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:19.437 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:19 smithi033 ceph-mon[27552]: from='osd.1 ' entity='osd.1' 2024-03-12T03:21:19.547 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:19 smithi170 ceph-mon[27708]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-12T03:21:19.547 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:19 smithi170 ceph-mon[27708]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T03:21:19.547 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:19 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:19.547 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:19 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:19.547 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:19 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:19.547 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:19 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:21:19.547 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:19 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:19.547 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:19 smithi170 ceph-mon[27708]: Adjusting osd_memory_target on smithi033 to 16069M 2024-03-12T03:21:19.547 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:19 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:19.547 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:19 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:19.547 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:19 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:19.548 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:19 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:19.548 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:19 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:19.548 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:19 smithi170 ceph-mon[27708]: from='osd.1 ' entity='osd.1' 2024-03-12T03:21:19.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:19 smithi007 ceph-mon[25800]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-12T03:21:19.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:19 smithi007 ceph-mon[25800]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T03:21:19.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:19.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:19.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:19.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:21:19.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:19.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:19 smithi007 ceph-mon[25800]: Adjusting osd_memory_target on smithi033 to 16069M 2024-03-12T03:21:19.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:19.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:19.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:19.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:19.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:19 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:19.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:19 smithi007 ceph-mon[25800]: from='osd.1 ' entity='osd.1' 2024-03-12T03:21:20.518 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:20 smithi170 ceph-mon[27708]: purged_snaps scrub starts 2024-03-12T03:21:20.518 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:20 smithi170 ceph-mon[27708]: purged_snaps scrub ok 2024-03-12T03:21:20.518 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:20 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:20.518 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:20 smithi170 ceph-mon[27708]: osd.1 [v2:172.21.15.33:6800/3621746125,v1:172.21.15.33:6801/3621746125] boot 2024-03-12T03:21:20.518 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:20 smithi170 ceph-mon[27708]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T03:21:20.518 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:20 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:20.519 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:20 smithi170 ceph-mon[27708]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:20.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:20 smithi033 ceph-mon[27552]: purged_snaps scrub starts 2024-03-12T03:21:20.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:20 smithi033 ceph-mon[27552]: purged_snaps scrub ok 2024-03-12T03:21:20.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:20 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:20.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:20 smithi033 ceph-mon[27552]: osd.1 [v2:172.21.15.33:6800/3621746125,v1:172.21.15.33:6801/3621746125] boot 2024-03-12T03:21:20.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:20 smithi033 ceph-mon[27552]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T03:21:20.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:20 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:20.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:20 smithi033 ceph-mon[27552]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:20.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:20 smithi007 ceph-mon[25800]: purged_snaps scrub starts 2024-03-12T03:21:20.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:20 smithi007 ceph-mon[25800]: purged_snaps scrub ok 2024-03-12T03:21:20.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:20 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:20.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:20 smithi007 ceph-mon[25800]: osd.1 [v2:172.21.15.33:6800/3621746125,v1:172.21.15.33:6801/3621746125] boot 2024-03-12T03:21:20.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:20 smithi007 ceph-mon[25800]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T03:21:20.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:20 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:21:20.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:20 smithi007 ceph-mon[25800]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:21:21.349 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/nvme4n1 2024-03-12T03:21:21.349 INFO:teuthology.orchestra.run.smithi170.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-12T03:21:21.349 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-12T03:21:21.349 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-12T03:21:21.349 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-12T03:21:21.349 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00935797 s, 1.1 GB/s 2024-03-12T03:21:21.349 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-12T03:21:21.982 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:21 smithi170 ceph-mon[27708]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T03:21:22.010 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi170:/dev/nvme4n1 2024-03-12T03:21:22.245 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T03:21:22.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:21 smithi033 ceph-mon[27552]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T03:21:22.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:21 smithi007 ceph-mon[25800]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T03:21:22.963 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:22 smithi170 ceph-mon[27708]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:23.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:22 smithi007 ceph-mon[25800]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:23.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:22 smithi033 ceph-mon[27552]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:24.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:23 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:21:24.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:23 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:21:24.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:23 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:24.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:23 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:21:24.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:23 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:21:24.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:23 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:24.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:23 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:21:24.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:23 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:21:24.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:23 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:25.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:24 smithi170 ceph-mon[27708]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:21:25.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:24 smithi170 ceph-mon[27708]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:25.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:24 smithi033 ceph-mon[27552]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:21:25.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:24 smithi033 ceph-mon[27552]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:25.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:24 smithi007 ceph-mon[25800]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:21:25.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:24 smithi007 ceph-mon[25800]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:26.497 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:26 smithi007 ceph-mon[25800]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:26.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:26 smithi033 ceph-mon[27552]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:26.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:26 smithi170 ceph-mon[27708]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:27.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:27 smithi033 ceph-mon[27552]: from='client.? 172.21.15.170:0/2517129506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8533b7d3-a661-4cc0-ad46-e7ab0faaa107"}]: dispatch 2024-03-12T03:21:27.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:27 smithi033 ceph-mon[27552]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8533b7d3-a661-4cc0-ad46-e7ab0faaa107"}]: dispatch 2024-03-12T03:21:27.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:27 smithi033 ceph-mon[27552]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8533b7d3-a661-4cc0-ad46-e7ab0faaa107"}]': finished 2024-03-12T03:21:27.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:27 smithi033 ceph-mon[27552]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T03:21:27.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:27 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:27.534 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:27 smithi007 ceph-mon[25800]: from='client.? 172.21.15.170:0/2517129506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8533b7d3-a661-4cc0-ad46-e7ab0faaa107"}]: dispatch 2024-03-12T03:21:27.534 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:27 smithi007 ceph-mon[25800]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8533b7d3-a661-4cc0-ad46-e7ab0faaa107"}]: dispatch 2024-03-12T03:21:27.534 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:27 smithi007 ceph-mon[25800]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8533b7d3-a661-4cc0-ad46-e7ab0faaa107"}]': finished 2024-03-12T03:21:27.534 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:27 smithi007 ceph-mon[25800]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T03:21:27.534 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:27 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:27.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:27 smithi170 ceph-mon[27708]: from='client.? 172.21.15.170:0/2517129506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8533b7d3-a661-4cc0-ad46-e7ab0faaa107"}]: dispatch 2024-03-12T03:21:27.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:27 smithi170 ceph-mon[27708]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8533b7d3-a661-4cc0-ad46-e7ab0faaa107"}]: dispatch 2024-03-12T03:21:27.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:27 smithi170 ceph-mon[27708]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8533b7d3-a661-4cc0-ad46-e7ab0faaa107"}]': finished 2024-03-12T03:21:27.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:27 smithi170 ceph-mon[27708]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T03:21:27.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:27 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:28.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:28 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:28.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:28 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:28.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:28 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:28.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:28 smithi007 ceph-mon[25800]: from='client.? 172.21.15.170:0/2911515892' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:21:28.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:28 smithi007 ceph-mon[25800]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:28.762 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:28 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:28.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:28 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:28.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:28 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:28.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:28 smithi033 ceph-mon[27552]: from='client.? 172.21.15.170:0/2911515892' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:21:28.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:28 smithi033 ceph-mon[27552]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:28.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:28 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:28.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:28 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:28.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:28 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:28.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:28 smithi170 ceph-mon[27708]: from='client.? 172.21.15.170:0/2911515892' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:21:28.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:28 smithi170 ceph-mon[27708]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:30.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:30 smithi170 ceph-mon[27708]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:30.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:30 smithi033 ceph-mon[27552]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:30.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:30 smithi007 ceph-mon[25800]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:32.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:32 smithi033 ceph-mon[27552]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:32.557 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:32 smithi170 ceph-mon[27708]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:32.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:32 smithi007 ceph-mon[25800]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:34.276 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:34 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:34.277 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:34 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:34.277 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:34 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:34.467 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:34 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:34.467 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:34 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:34.467 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:34 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:34.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:34 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:34.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:34 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:34.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:34 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:35.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:35 smithi033 ceph-mon[27552]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:35.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:35 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:35.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:35 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:35.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:35 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:35.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:35 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T03:21:35.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:35 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:35.581 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:35 smithi170 ceph-mon[27708]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:35.581 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:35 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:35.581 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:35 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:35.582 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:35 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:35.582 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:35 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T03:21:35.582 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:35 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:35.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:35 smithi007 ceph-mon[25800]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:35.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:35 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:35.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:35 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:35.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:35 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:35.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:35 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T03:21:35.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:35 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:36.485 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:36 smithi170 ceph-mon[27708]: Detected new or changed devices on smithi033 2024-03-12T03:21:36.485 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:36 smithi170 ceph-mon[27708]: Deploying daemon osd.2 on smithi170 2024-03-12T03:21:36.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:36 smithi033 ceph-mon[27552]: Detected new or changed devices on smithi033 2024-03-12T03:21:36.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:36 smithi033 ceph-mon[27552]: Deploying daemon osd.2 on smithi170 2024-03-12T03:21:36.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:36 smithi007 ceph-mon[25800]: Detected new or changed devices on smithi033 2024-03-12T03:21:36.617 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:36 smithi007 ceph-mon[25800]: Deploying daemon osd.2 on smithi170 2024-03-12T03:21:37.213 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:37 smithi170 ceph-mon[27708]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:37.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:37 smithi033 ceph-mon[27552]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:37.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:37 smithi007 ceph-mon[25800]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:38.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:38 smithi033 ceph-mon[27552]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:38.470 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:38 smithi170 ceph-mon[27708]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:38.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:38 smithi007 ceph-mon[25800]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:40.336 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:40 smithi170 ceph-mon[27708]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:40.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:40 smithi033 ceph-mon[27552]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:40.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:40 smithi007 ceph-mon[25800]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:42.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:42 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:42.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:42 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:42.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:42 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:42.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:42 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:42.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:42 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:42.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:42 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:42.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:42 smithi033 ceph-mon[27552]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:42.595 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:42 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:42.595 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:42 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:42.595 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:42 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:42.595 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:42 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:42.595 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:42 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:42.595 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:42 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:42.595 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:42 smithi170 ceph-mon[27708]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:42.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:42 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:42.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:42 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:42.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:42 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:42.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:42 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:42.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:42 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:42.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:42 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:42.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:42 smithi007 ceph-mon[25800]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:44.331 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:44 smithi170 ceph-mon[27708]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:44.375 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 2 on host 'smithi170' 2024-03-12T03:21:44.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:44 smithi033 ceph-mon[27552]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:44.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:44 smithi007 ceph-mon[25800]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:44.979 DEBUG:teuthology.orchestra.run.smithi170:osd.2> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@osd.2.service 2024-03-12T03:21:44.981 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-12T03:21:44.982 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-12T03:21:45.236 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:21:45.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:45 smithi007 ceph-mon[25800]: from='osd.2 [v2:172.21.15.170:6800/3615959269,v1:172.21.15.170:6801/3615959269]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:21:45.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:45 smithi007 ceph-mon[25800]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:21:45.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:45 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:45.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:45 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:45.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:45 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:45.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:45 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:21:45.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:45 smithi007 ceph-mon[25800]: Adjusting osd_memory_target on smithi170 to 16064M 2024-03-12T03:21:45.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:45 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:45.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:45 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:45.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:45 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:45.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:45 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:45.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:45 smithi033 ceph-mon[27552]: from='osd.2 [v2:172.21.15.170:6800/3615959269,v1:172.21.15.170:6801/3615959269]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:21:45.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:45 smithi033 ceph-mon[27552]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:21:45.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:45 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:45.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:45 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:45.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:45 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:45.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:45 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:21:45.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:45 smithi033 ceph-mon[27552]: Adjusting osd_memory_target on smithi170 to 16064M 2024-03-12T03:21:45.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:45 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:45.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:45 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:45.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:45 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:45.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:45 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:45.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:45 smithi170 ceph-mon[27708]: from='osd.2 [v2:172.21.15.170:6800/3615959269,v1:172.21.15.170:6801/3615959269]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:21:45.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:45 smithi170 ceph-mon[27708]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:21:45.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:45 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:45.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:45 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:45.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:45 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:45.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:45 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:21:45.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:45 smithi170 ceph-mon[27708]: Adjusting osd_memory_target on smithi170 to 16064M 2024-03-12T03:21:45.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:45 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:45.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:45 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:45.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:45 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:45.614 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:45 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:46.522 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:46 smithi007 ceph-mon[25800]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T03:21:46.522 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:46 smithi007 ceph-mon[25800]: from='osd.2 [v2:172.21.15.170:6800/3615959269,v1:172.21.15.170:6801/3615959269]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-12T03:21:46.522 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:46 smithi007 ceph-mon[25800]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T03:21:46.522 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:46 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:46.522 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:46 smithi007 ceph-mon[25800]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-12T03:21:46.522 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:46 smithi007 ceph-mon[25800]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:46.537 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:46 smithi170 ceph-mon[27708]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T03:21:46.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:46 smithi170 ceph-mon[27708]: from='osd.2 [v2:172.21.15.170:6800/3615959269,v1:172.21.15.170:6801/3615959269]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-12T03:21:46.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:46 smithi170 ceph-mon[27708]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T03:21:46.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:46 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:46.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:46 smithi170 ceph-mon[27708]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-12T03:21:46.538 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:46 smithi170 ceph-mon[27708]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:46.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:46 smithi033 ceph-mon[27552]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T03:21:46.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:46 smithi033 ceph-mon[27552]: from='osd.2 [v2:172.21.15.170:6800/3615959269,v1:172.21.15.170:6801/3615959269]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-12T03:21:46.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:46 smithi033 ceph-mon[27552]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T03:21:46.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:46 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:46.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:46 smithi033 ceph-mon[27552]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-12T03:21:46.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:46 smithi033 ceph-mon[27552]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:46.863 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:21:46 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-2[34306]: 2024-03-12T03:21:46.533+0000 7f8fd4648700 -1 osd.2 0 waiting for initial osdmap 2024-03-12T03:21:46.863 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:21:46 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-2[34306]: 2024-03-12T03:21:46.536+0000 7f8fcc433700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:21:47.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:47 smithi007 ceph-mon[25800]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-12T03:21:47.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:47 smithi007 ceph-mon[25800]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T03:21:47.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:47 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:47.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:47 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:47.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:47 smithi007 ceph-mon[25800]: from='osd.2 ' entity='osd.2' 2024-03-12T03:21:47.505 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:21:47.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:47 smithi033 ceph-mon[27552]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-12T03:21:47.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:47 smithi033 ceph-mon[27552]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T03:21:47.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:47 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:47.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:47 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:47.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:47 smithi033 ceph-mon[27552]: from='osd.2 ' entity='osd.2' 2024-03-12T03:21:47.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:47 smithi170 ceph-mon[27708]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-12T03:21:47.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:47 smithi170 ceph-mon[27708]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T03:21:47.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:47 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:47.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:47 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:47.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:47 smithi170 ceph-mon[27708]: from='osd.2 ' entity='osd.2' 2024-03-12T03:21:48.381 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":17,"num_osds":3,"num_up_osds":2,"osd_up_since":1710213679,"num_in_osds":3,"osd_in_since":1710213686,"num_remapped_pgs":0} 2024-03-12T03:21:48.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:48 smithi007 ceph-mon[25800]: purged_snaps scrub starts 2024-03-12T03:21:48.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:48 smithi007 ceph-mon[25800]: purged_snaps scrub ok 2024-03-12T03:21:48.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:48 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:48.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:48 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/1754275785' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:21:48.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:48 smithi007 ceph-mon[25800]: osd.2 [v2:172.21.15.170:6800/3615959269,v1:172.21.15.170:6801/3615959269] boot 2024-03-12T03:21:48.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:48 smithi007 ceph-mon[25800]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T03:21:48.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:48 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:48.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:48 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:48.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:48 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:48.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:48 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:48.635 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:48 smithi007 ceph-mon[25800]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:48.636 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:48 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:48 smithi033 ceph-mon[27552]: purged_snaps scrub starts 2024-03-12T03:21:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:48 smithi033 ceph-mon[27552]: purged_snaps scrub ok 2024-03-12T03:21:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:48 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:48 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/1754275785' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:21:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:48 smithi033 ceph-mon[27552]: osd.2 [v2:172.21.15.170:6800/3615959269,v1:172.21.15.170:6801/3615959269] boot 2024-03-12T03:21:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:48 smithi033 ceph-mon[27552]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T03:21:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:48 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:48 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:48.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:48 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:48.764 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:48 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:48.764 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:48 smithi033 ceph-mon[27552]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:48.764 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:48 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:48.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:48 smithi170 ceph-mon[27708]: purged_snaps scrub starts 2024-03-12T03:21:48.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:48 smithi170 ceph-mon[27708]: purged_snaps scrub ok 2024-03-12T03:21:48.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:48 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:48 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/1754275785' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:21:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:48 smithi170 ceph-mon[27708]: osd.2 [v2:172.21.15.170:6800/3615959269,v1:172.21.15.170:6801/3615959269] boot 2024-03-12T03:21:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:48 smithi170 ceph-mon[27708]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T03:21:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:48 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:21:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:48 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:48 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:48 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:48 smithi170 ceph-mon[27708]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:21:48.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:48 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:49.382 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-12T03:21:49.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:49 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:49.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:49 smithi007 ceph-mon[25800]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T03:21:49.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:49 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:49.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:49 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:49.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:49 smithi170 ceph-mon[27708]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T03:21:49.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:49 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:50.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:49 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:50.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:49 smithi033 ceph-mon[27552]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T03:21:50.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:49 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:21:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:21:50.179 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:21:49 smithi007 sudo[40265]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-12T03:21:50.179 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:21:49 smithi007 sudo[40265]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:21:50.179 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:21:49 smithi007 sudo[40265]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:21:50.179 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:21:49 smithi007 sudo[40265]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:21:50.449 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:21:50 smithi007 sudo[40312]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-12T03:21:50.449 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:21:50 smithi007 sudo[40312]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:21:50.449 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:21:50 smithi007 sudo[40312]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:21:50.449 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:21:50 smithi007 sudo[40312]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:21:50.735 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:50 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:50.735 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:50 smithi033 ceph-mon[27552]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T03:21:50.736 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:50 smithi033 ceph-mon[27552]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:21:50.736 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:50 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:50.736 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:21:50 smithi033 sudo[34405]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-12T03:21:50.736 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:21:50 smithi033 sudo[34405]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:21:50.736 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:21:50 smithi033 sudo[34405]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:21:50.737 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:21:50 smithi033 sudo[34405]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:21:50.850 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:50 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:50.850 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:50 smithi007 ceph-mon[25800]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T03:21:50.850 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:50 smithi007 ceph-mon[25800]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:21:50.850 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:50 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:50.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:50 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:21:50.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:50 smithi170 ceph-mon[27708]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T03:21:50.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:50 smithi170 ceph-mon[27708]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:21:50.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:50 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:51.013 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:21:50 smithi033 sudo[34408]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-12T03:21:51.013 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:21:50 smithi033 sudo[34408]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:21:51.013 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:21:50 smithi033 sudo[34408]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:21:51.013 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:21:50 smithi033 sudo[34408]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:21:51.250 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:21:50 smithi170 sudo[35190]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-12T03:21:51.250 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:21:50 smithi170 sudo[35190]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:21:51.250 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:21:50 smithi170 sudo[35190]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:21:51.250 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:21:51 smithi170 sudo[35190]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:21:51.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:51 smithi007 sudo[40401]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:21:51.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:51 smithi007 sudo[40401]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:21:51.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:51 smithi007 sudo[40401]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:21:51.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:51 smithi007 sudo[40401]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:21:51.613 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:21:51 smithi170 sudo[35193]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-12T03:21:51.613 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:21:51 smithi170 sudo[35193]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:21:51.613 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:21:51 smithi170 sudo[35193]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:21:51.613 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:21:51 smithi170 sudo[35193]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:21:51.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:51 smithi007 sudo[40429]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:21:51.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:51 smithi007 sudo[40429]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:21:51.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:51 smithi007 sudo[40429]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:21:51.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:51 smithi007 sudo[40429]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:21:51.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:51 smithi007 ceph-mon[25800]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T03:21:51.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:51 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:51.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:51 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:51.849 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:51 smithi007 ceph-mon[25800]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:21:52.027 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:21:52.067 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:51 smithi170 ceph-mon[27708]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T03:21:52.068 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:51 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:52.068 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:51 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:52.068 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:51 smithi170 ceph-mon[27708]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:21:52.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:51 smithi033 ceph-mon[27552]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T03:21:52.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:51 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:52.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:51 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:52.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:51 smithi033 ceph-mon[27552]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:21:52.319 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 sudo[35196]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:21:52.319 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 sudo[35196]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:21:52.319 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 sudo[35196]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:21:52.319 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 sudo[35196]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:21:52.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 sudo[35199]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:21:52.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 sudo[35199]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:21:52.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 sudo[35199]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:21:52.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 sudo[35199]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:21:52.705 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"num_osds":3,"num_up_osds":3,"osd_up_since":1710213707,"num_in_osds":3,"osd_in_since":1710213686,"num_remapped_pgs":0} 2024-03-12T03:21:52.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:21:52.867 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 sudo[34411]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:21:52.867 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 sudo[34411]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:21:52.867 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 sudo[34411]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:21:52.867 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 sudo[34411]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:21:52.948 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:21:53.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:21:53.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:21:53.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:21:53.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:21:53.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-12T03:21:53.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:53.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:21:53.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:21:53.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:21:53.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:21:53.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/2408911410' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:21:53.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 93 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:21:53.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:21:53.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:53.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:21:53.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:21:53.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:52 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:21:53.143 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:21:53.144 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:21:53.144 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:21:53.144 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:21:53.144 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-12T03:21:53.144 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:53.144 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:21:53.144 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:21:53.144 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:21:53.144 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:21:53.144 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/2408911410' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:21:53.144 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 93 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:21:53.144 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:21:53.144 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:53.145 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:21:53.145 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:21:53.145 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:52 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:21:53.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 sudo[34414]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:21:53.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 sudo[34414]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:21:53.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 sudo[34414]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:21:53.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 sudo[34414]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:21:53.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:21:53.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:21:53.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:21:53.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:21:53.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-12T03:21:53.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:53.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:21:53.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:21:53.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:21:53.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:21:53.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/2408911410' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:21:53.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 93 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:21:53.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:21:53.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:53.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:21:53.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:21:53.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:52 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:21:54.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:21:54.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:21:54.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:54.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:54.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:54.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:54.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:54.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:54.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:53 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:21:54.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:21:54.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:54.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:54.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:54.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:54.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:54.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:54.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:53 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:21:54.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:21:54.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:54.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:54.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:54.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:54.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:54.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:54.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:53 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:54.752 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:21:54.752 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"fsid":"fe67b1e4-e01e-11ee-95c8-87774f69a715","created":"2024-03-12T03:18:17.656051+0000","modified":"2024-03-12T03:21:50.583127+0000","last_up_change":"2024-03-12T03:21:47.536306+0000","last_in_change":"2024-03-12T03:21:26.563938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:21:48.477722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9444b255-7dd8-476c-a981-596313fa6856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":4033945070},{"type":"v1","addr":"172.21.15.7:6803","nonce":4033945070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":4033945070},{"type":"v1","addr":"172.21.15.7:6805","nonce":4033945070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":4033945070},{"type":"v1","addr":"172.21.15.7:6809","nonce":4033945070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":4033945070},{"type":"v1","addr":"172.21.15.7:6807","nonce":4033945070}]},"public_addr":"172.21.15.7:6803/4033945070","cluster_addr":"172.21.15.7:6805/4033945070","heartbeat_back_addr":"172.21.15.7:6809/4033945070","heartbeat_front_addr":"172.21.15.7:6807/4033945070","state":["exists","up"]},{"osd":1,"uuid":"06e71f2a-abc2-43f6-af2c-67eb8cba66f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":3621746125},{"type":"v1","addr":"172.21.15.33:6801","nonce":3621746125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":3621746125},{"type":"v1","addr":"172.21.15.33:6803","nonce":3621746125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":3621746125},{"type":"v1","addr":"172.21.15.33:6807","nonce":3621746125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":3621746125},{"type":"v1","addr":"172.21.15.33:6805","nonce":3621746125}]},"public_addr":"172.21.15.33:6801/3621746125","cluster_addr":"172.21.15.33:6803/3621746125","heartbeat_back_addr":"172.21.15.33:6807/3621746125","heartbeat_front_addr":"172.21.15.33:6805/3621746125","state":["exists","up"]},{"osd":2,"uuid":"8533b7d3-a661-4cc0-ad46-e7ab0faaa107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3615959269},{"type":"v1","addr":"172.21.15.170:6801","nonce":3615959269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3615959269},{"type":"v1","addr":"172.21.15.170:6803","nonce":3615959269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3615959269},{"type":"v1","addr":"172.21.15.170:6807","nonce":3615959269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3615959269},{"type":"v1","addr":"172.21.15.170:6805","nonce":3615959269}]},"public_addr":"172.21.15.170:6801/3615959269","cluster_addr":"172.21.15.170:6803/3615959269","heartbeat_back_addr":"172.21.15.170:6807/3615959269","heartbeat_front_addr":"172.21.15.170:6805/3615959269","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:20:51.583068+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:21:17.997089+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:21:45.317595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/821359088":"2024-03-13T03:19:16.129168+0000","172.21.15.7:0/3516588619":"2024-03-13T03:19:16.129168+0000","172.21.15.7:6800/1100758856":"2024-03-13T03:18:49.567015+0000","172.21.15.7:0/3508574268":"2024-03-13T03:19:16.129168+0000","172.21.15.7:6800/928905743":"2024-03-13T03:19:16.129168+0000","172.21.15.7:0/3944002862":"2024-03-13T03:18:49.567015+0000","172.21.15.7:6801/928905743":"2024-03-13T03:19:16.129168+0000","172.21.15.7:6801/1100758856":"2024-03-13T03:18:49.567015+0000","172.21.15.7:0/2116937091":"2024-03-13T03:18:49.567015+0000","172.21.15.7:0/503027408":"2024-03-13T03:18:49.567015+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:21:55.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:54 smithi007 ceph-mon[25800]: Detected new or changed devices on smithi170 2024-03-12T03:21:55.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:54 smithi007 ceph-mon[25800]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:21:55.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:54 smithi007 ceph-mon[25800]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:21:55.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:54 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:55.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:54 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:55.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:54 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:55.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:54 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/2005496884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:21:55.247 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:54 smithi170 ceph-mon[27708]: Detected new or changed devices on smithi170 2024-03-12T03:21:55.247 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:54 smithi170 ceph-mon[27708]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:21:55.247 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:54 smithi170 ceph-mon[27708]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:21:55.247 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:54 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:55.247 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:54 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:55.247 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:54 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:55.247 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:54 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/2005496884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:21:55.262 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:54 smithi033 ceph-mon[27552]: Detected new or changed devices on smithi170 2024-03-12T03:21:55.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:54 smithi033 ceph-mon[27552]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:21:55.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:54 smithi033 ceph-mon[27552]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:21:55.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:54 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:55.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:54 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:55.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:54 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:21:55.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:54 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/2005496884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:21:55.459 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T03:21:48.477722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T03:21:55.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-12T03:21:55.709 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:21:57.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:56 smithi007 ceph-mon[25800]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 565 KiB/s wr, 36 op/s 2024-03-12T03:21:57.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:56 smithi170 ceph-mon[27708]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 565 KiB/s wr, 36 op/s 2024-03-12T03:21:57.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:56 smithi033 ceph-mon[27552]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 565 KiB/s wr, 36 op/s 2024-03-12T03:21:57.554 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-12T03:21:58.001 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:57 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/1107317581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:21:58.153 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:57 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/1107317581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:21:58.184 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi007 2024-03-12T03:21:58.185 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi007=foo.a' 2024-03-12T03:21:58.203 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:57 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/1107317581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:21:58.443 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T03:21:59.258 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:21:58 smithi170 ceph-mon[27708]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 7.6 KiB/s rd, 406 KiB/s wr, 25 op/s 2024-03-12T03:21:59.262 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:21:58 smithi033 ceph-mon[27552]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 7.6 KiB/s rd, 406 KiB/s wr, 25 op/s 2024-03-12T03:21:59.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:21:58 smithi007 ceph-mon[25800]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 7.6 KiB/s rd, 406 KiB/s wr, 25 op/s 2024-03-12T03:22:00.296 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled rgw.foo.a update... 2024-03-12T03:22:00.902 DEBUG:teuthology.orchestra.run.smithi007:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-12T03:22:00.905 INFO:tasks.cephadm:Adding prometheus.a on smithi033 2024-03-12T03:22:00.905 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi033=a' 2024-03-12T03:22:01.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:00 smithi007 ceph-mon[25800]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 5.9 KiB/s rd, 316 KiB/s wr, 20 op/s 2024-03-12T03:22:01.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:00 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:01.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:00 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:01.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:00 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:01.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:00 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:01.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:00 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:01.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:00 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:22:01.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:00 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T03:22:01.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:00 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:01.139 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:00 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:01.176 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T03:22:01.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:00 smithi033 ceph-mon[27552]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 5.9 KiB/s rd, 316 KiB/s wr, 20 op/s 2024-03-12T03:22:01.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:00 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:01.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:00 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:01.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:00 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:01.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:00 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:01.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:00 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:01.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:00 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:22:01.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:00 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T03:22:01.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:00 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:01.264 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:00 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:01.297 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:00 smithi170 ceph-mon[27708]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 5.9 KiB/s rd, 316 KiB/s wr, 20 op/s 2024-03-12T03:22:01.297 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:00 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:01.297 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:00 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:01.297 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:00 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:01.297 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:00 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:01.297 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:00 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:01.297 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:00 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:22:01.297 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:00 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T03:22:01.297 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:00 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:01.297 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:00 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:02.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:01 smithi007 ceph-mon[25800]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi007=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:02.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:01 smithi007 ceph-mon[25800]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-03-12T03:22:02.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:01 smithi007 ceph-mon[25800]: Deploying daemon rgw.foo.a on smithi007 2024-03-12T03:22:02.171 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:01 smithi170 ceph-mon[27708]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi007=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:02.171 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:01 smithi170 ceph-mon[27708]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-03-12T03:22:02.171 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:01 smithi170 ceph-mon[27708]: Deploying daemon rgw.foo.a on smithi007 2024-03-12T03:22:02.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:01 smithi033 ceph-mon[27552]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi007=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:02.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:01 smithi033 ceph-mon[27552]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-03-12T03:22:02.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:01 smithi033 ceph-mon[27552]: Deploying daemon rgw.foo.a on smithi007 2024-03-12T03:22:02.598 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:22:02 smithi007 systemd[1]: Starting Ceph rgw.foo.a for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:22:02.966 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:22:02 smithi007 podman[41257]: 2024-03-12 03:22:02.601073323 +0000 UTC m=+0.113959963 container create 6018bc187853096a38bd584e6076b5b033ac9bd35976567a0647ec36e42e67c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-12T03:22:02.966 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:22:02 smithi007 podman[41257]: 2024-03-12 03:22:02.505380388 +0000 UTC m=+0.018266965 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:22:02.966 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:22:02 smithi007 podman[41257]: 2024-03-12 03:22:02.845580661 +0000 UTC m=+0.358467248 container init 6018bc187853096a38bd584e6076b5b033ac9bd35976567a0647ec36e42e67c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-12T03:22:02.966 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:22:02 smithi007 podman[41257]: 2024-03-12 03:22:02.85069315 +0000 UTC m=+0.363579725 container start 6018bc187853096a38bd584e6076b5b033ac9bd35976567a0647ec36e42e67c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-12T03:22:02.966 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:22:02 smithi007 bash[41257]: 6018bc187853096a38bd584e6076b5b033ac9bd35976567a0647ec36e42e67c3 2024-03-12T03:22:03.139 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled prometheus update... 2024-03-12T03:22:03.348 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:22:02 smithi007 systemd[1]: Started Ceph rgw.foo.a for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:22:03.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:03 smithi007 ceph-mon[25800]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 5.7 KiB/s rd, 302 KiB/s wr, 19 op/s 2024-03-12T03:22:03.459 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:03 smithi170 ceph-mon[27708]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 5.7 KiB/s rd, 302 KiB/s wr, 19 op/s 2024-03-12T03:22:03.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:03 smithi033 ceph-mon[27552]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 5.7 KiB/s rd, 302 KiB/s wr, 19 op/s 2024-03-12T03:22:03.762 DEBUG:teuthology.orchestra.run.smithi033:prometheus.a> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-12T03:22:03.765 INFO:tasks.cephadm:Adding node-exporter.a on smithi007 2024-03-12T03:22:03.765 INFO:tasks.cephadm:Adding node-exporter.b on smithi033 2024-03-12T03:22:03.765 INFO:tasks.cephadm:Adding node-exporter.c on smithi170 2024-03-12T03:22:03.765 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '3;smithi007=a;smithi033=b;smithi170=c' 2024-03-12T03:22:03.997 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T03:22:04.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:04 smithi007 ceph-mon[25800]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi033=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:04.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:04 smithi007 ceph-mon[25800]: Saving service prometheus spec with placement smithi033=a;count:1 2024-03-12T03:22:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:04 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:04 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:04 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:04 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:04 smithi007 ceph-mon[25800]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-03-12T03:22:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:04 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:04 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:04 smithi007 ceph-mon[25800]: osdmap e22: 3 total, 3 up, 3 in 2024-03-12T03:22:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:04 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/4104014568' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T03:22:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:04 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:04 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:04.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:04 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:04.455 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:04 smithi170 ceph-mon[27708]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi033=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:04.455 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:04 smithi170 ceph-mon[27708]: Saving service prometheus spec with placement smithi033=a;count:1 2024-03-12T03:22:04.455 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:04 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.455 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:04 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.455 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:04 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.455 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:04 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.455 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:04 smithi170 ceph-mon[27708]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-03-12T03:22:04.455 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:04 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.455 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:04 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.456 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:04 smithi170 ceph-mon[27708]: osdmap e22: 3 total, 3 up, 3 in 2024-03-12T03:22:04.456 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:04 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/4104014568' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T03:22:04.456 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:04 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:04.456 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:04 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:04.456 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:04 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:04.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:04 smithi033 ceph-mon[27552]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi033=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:04.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:04 smithi033 ceph-mon[27552]: Saving service prometheus spec with placement smithi033=a;count:1 2024-03-12T03:22:04.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:04 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:04 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:04 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:04 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:04 smithi033 ceph-mon[27552]: Saving service rgw.foo.a spec with placement smithi007=foo.a;count:1 2024-03-12T03:22:04.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:04 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:04 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:04.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:04 smithi033 ceph-mon[27552]: osdmap e22: 3 total, 3 up, 3 in 2024-03-12T03:22:04.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:04 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/4104014568' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T03:22:04.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:04 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:04.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:04 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:04.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:04 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:05.209 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:05 smithi170 ceph-mon[27708]: Checking dashboard <-> RGW credentials 2024-03-12T03:22:05.209 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:05 smithi170 ceph-mon[27708]: pgmap v97: 33 pgs: 14 creating+peering, 1 active+clean, 18 unknown; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 5.4 KiB/s rd, 287 KiB/s wr, 18 op/s 2024-03-12T03:22:05.209 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:05 smithi170 ceph-mon[27708]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:22:05.209 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:05 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/4104014568' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-12T03:22:05.209 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:05 smithi170 ceph-mon[27708]: osdmap e23: 3 total, 3 up, 3 in 2024-03-12T03:22:05.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:05 smithi033 ceph-mon[27552]: Checking dashboard <-> RGW credentials 2024-03-12T03:22:05.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:05 smithi033 ceph-mon[27552]: pgmap v97: 33 pgs: 14 creating+peering, 1 active+clean, 18 unknown; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 5.4 KiB/s rd, 287 KiB/s wr, 18 op/s 2024-03-12T03:22:05.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:05 smithi033 ceph-mon[27552]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:22:05.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:05 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/4104014568' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-12T03:22:05.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:05 smithi033 ceph-mon[27552]: osdmap e23: 3 total, 3 up, 3 in 2024-03-12T03:22:05.516 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:05 smithi007 ceph-mon[25800]: Checking dashboard <-> RGW credentials 2024-03-12T03:22:05.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:05 smithi007 ceph-mon[25800]: pgmap v97: 33 pgs: 14 creating+peering, 1 active+clean, 18 unknown; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 5.4 KiB/s rd, 287 KiB/s wr, 18 op/s 2024-03-12T03:22:05.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:05 smithi007 ceph-mon[25800]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:22:05.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:05 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/4104014568' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-12T03:22:05.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:05 smithi007 ceph-mon[25800]: osdmap e23: 3 total, 3 up, 3 in 2024-03-12T03:22:05.926 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled node-exporter update... 2024-03-12T03:22:06.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:06 smithi007 ceph-mon[25800]: osdmap e24: 3 total, 3 up, 3 in 2024-03-12T03:22:06.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:06 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:22:06.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:06 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:22:06.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:06 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:06.481 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:06 smithi170 ceph-mon[27708]: osdmap e24: 3 total, 3 up, 3 in 2024-03-12T03:22:06.481 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:06 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:22:06.481 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:06 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:22:06.481 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:06 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:06.507 DEBUG:teuthology.orchestra.run.smithi007:node-exporter.a> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-12T03:22:06.510 DEBUG:teuthology.orchestra.run.smithi033:node-exporter.b> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-12T03:22:06.513 DEBUG:teuthology.orchestra.run.smithi170:node-exporter.c> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@node-exporter.c.service 2024-03-12T03:22:06.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:06 smithi033 ceph-mon[27552]: osdmap e24: 3 total, 3 up, 3 in 2024-03-12T03:22:06.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:06 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:22:06.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:06 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:22:06.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:06 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:06.516 INFO:tasks.cephadm:Adding alertmanager.a on smithi007 2024-03-12T03:22:06.516 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi007=a' 2024-03-12T03:22:06.786 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T03:22:07.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:07 smithi033 ceph-mon[27552]: pgmap v100: 65 pgs: 25 creating+peering, 19 active+clean, 21 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 682 B/s wr, 1 op/s 2024-03-12T03:22:07.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:07 smithi033 ceph-mon[27552]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi007=a;smithi033=b;smithi170=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:07.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:07 smithi033 ceph-mon[27552]: Saving service node-exporter spec with placement smithi007=a;smithi033=b;smithi170=c;count:3 2024-03-12T03:22:07.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:07 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:22:07.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:07 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:22:07.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:07 smithi033 ceph-mon[27552]: osdmap e25: 3 total, 3 up, 3 in 2024-03-12T03:22:07.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:07 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:07.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:07 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:07.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:07 smithi007 ceph-mon[25800]: pgmap v100: 65 pgs: 25 creating+peering, 19 active+clean, 21 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 682 B/s wr, 1 op/s 2024-03-12T03:22:07.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:07 smithi007 ceph-mon[25800]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi007=a;smithi033=b;smithi170=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:07.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:07 smithi007 ceph-mon[25800]: Saving service node-exporter spec with placement smithi007=a;smithi033=b;smithi170=c;count:3 2024-03-12T03:22:07.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:07 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:22:07.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:07 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:22:07.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:07 smithi007 ceph-mon[25800]: osdmap e25: 3 total, 3 up, 3 in 2024-03-12T03:22:07.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:07.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:07 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:07.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:07 smithi170 ceph-mon[27708]: pgmap v100: 65 pgs: 25 creating+peering, 19 active+clean, 21 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 682 B/s wr, 1 op/s 2024-03-12T03:22:07.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:07 smithi170 ceph-mon[27708]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi007=a;smithi033=b;smithi170=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:07.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:07 smithi170 ceph-mon[27708]: Saving service node-exporter spec with placement smithi007=a;smithi033=b;smithi170=c;count:3 2024-03-12T03:22:07.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:07 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:22:07.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:07 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-12T03:22:07.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:07 smithi170 ceph-mon[27708]: osdmap e25: 3 total, 3 up, 3 in 2024-03-12T03:22:07.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:07.613 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:07 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:08.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:08 smithi170 ceph-mon[27708]: osdmap e26: 3 total, 3 up, 3 in 2024-03-12T03:22:08.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:08 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:22:08.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:08 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:22:08.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:08 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:08.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:08 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:08.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:08 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:08.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:08 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:22:08.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:08 smithi033 ceph-mon[27552]: osdmap e26: 3 total, 3 up, 3 in 2024-03-12T03:22:08.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:08 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:22:08.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:08 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:22:08.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:08 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:08.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:08 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:08.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:08 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:08.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:08 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:22:08.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:08 smithi007 ceph-mon[25800]: osdmap e26: 3 total, 3 up, 3 in 2024-03-12T03:22:08.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:08 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:22:08.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:08 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:22:08.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:08 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:08.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:08 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:08.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:08 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:08.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:08 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:22:08.767 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled alertmanager update... 2024-03-12T03:22:09.361 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:09 smithi170 ceph-mon[27708]: pgmap v103: 97 pgs: 25 creating+peering, 19 active+clean, 53 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:22:09.361 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:09 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:22:09.361 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:09 smithi170 ceph-mon[27708]: osdmap e27: 3 total, 3 up, 3 in 2024-03-12T03:22:09.361 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:09 smithi170 ceph-mon[27708]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:09.361 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:09 smithi170 ceph-mon[27708]: Saving service alertmanager spec with placement smithi007=a;count:1 2024-03-12T03:22:09.361 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:09 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:09.387 DEBUG:teuthology.orchestra.run.smithi007:alertmanager.a> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-12T03:22:09.390 INFO:tasks.cephadm:Adding grafana.a on smithi033 2024-03-12T03:22:09.390 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi033=a' 2024-03-12T03:22:09.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:09 smithi033 ceph-mon[27552]: pgmap v103: 97 pgs: 25 creating+peering, 19 active+clean, 53 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:22:09.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:09 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:22:09.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:09 smithi033 ceph-mon[27552]: osdmap e27: 3 total, 3 up, 3 in 2024-03-12T03:22:09.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:09 smithi033 ceph-mon[27552]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:09.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:09 smithi033 ceph-mon[27552]: Saving service alertmanager spec with placement smithi007=a;count:1 2024-03-12T03:22:09.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:09 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:09.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:09 smithi007 ceph-mon[25800]: pgmap v103: 97 pgs: 25 creating+peering, 19 active+clean, 53 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:22:09.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:09 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-12T03:22:09.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:09 smithi007 ceph-mon[25800]: osdmap e27: 3 total, 3 up, 3 in 2024-03-12T03:22:09.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:09 smithi007 ceph-mon[25800]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:09.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:09 smithi007 ceph-mon[25800]: Saving service alertmanager spec with placement smithi007=a;count:1 2024-03-12T03:22:09.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:09 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:09.626 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T03:22:10.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:10 smithi170 ceph-mon[27708]: osdmap e28: 3 total, 3 up, 3 in 2024-03-12T03:22:10.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:10 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:22:10.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:10 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:22:10.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:10 smithi170 ceph-mon[27708]: pgmap v106: 129 pgs: 21 creating+peering, 77 active+clean, 31 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-12T03:22:10.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:10 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:22:10.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:10 smithi033 ceph-mon[27552]: osdmap e28: 3 total, 3 up, 3 in 2024-03-12T03:22:10.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:10 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:22:10.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:10 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:22:10.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:10 smithi033 ceph-mon[27552]: pgmap v106: 129 pgs: 21 creating+peering, 77 active+clean, 31 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-12T03:22:10.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:10 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:22:10.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:10 smithi007 ceph-mon[25800]: osdmap e28: 3 total, 3 up, 3 in 2024-03-12T03:22:10.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:10 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:22:10.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:10 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:22:10.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:10 smithi007 ceph-mon[25800]: pgmap v106: 129 pgs: 21 creating+peering, 77 active+clean, 31 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-12T03:22:10.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:10 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:22:11.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:11 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:22:11.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:11 smithi170 ceph-mon[27708]: osdmap e29: 3 total, 3 up, 3 in 2024-03-12T03:22:11.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:11 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:22:11.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:11 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:22:11.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:11 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:22:11.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:11 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:22:11.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:11 smithi170 ceph-mon[27708]: osdmap e30: 3 total, 3 up, 3 in 2024-03-12T03:22:11.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:11 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:22:11.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:11 smithi033 ceph-mon[27552]: osdmap e29: 3 total, 3 up, 3 in 2024-03-12T03:22:11.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:11 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:22:11.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:11 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:22:11.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:11 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:22:11.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:11 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:22:11.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:11 smithi033 ceph-mon[27552]: osdmap e30: 3 total, 3 up, 3 in 2024-03-12T03:22:11.538 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled grafana update... 2024-03-12T03:22:11.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:11 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:22:11.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:11 smithi007 ceph-mon[25800]: osdmap e29: 3 total, 3 up, 3 in 2024-03-12T03:22:11.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:11 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:22:11.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:11 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:22:11.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:11 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/152978430' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:22:11.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:11 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/3562802868' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:22:11.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:11 smithi007 ceph-mon[25800]: osdmap e30: 3 total, 3 up, 3 in 2024-03-12T03:22:12.158 DEBUG:teuthology.orchestra.run.smithi033:grafana.a> sudo journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@grafana.a.service 2024-03-12T03:22:12.160 INFO:tasks.cephadm:Setting up client nodes... 2024-03-12T03:22:12.161 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T03:22:12.412 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:22:12.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:12 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:12.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:12 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:12.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:12 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:12.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:12 smithi033 ceph-mon[27552]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi033=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:12.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:12 smithi033 ceph-mon[27552]: Saving service grafana spec with placement smithi033=a;count:1 2024-03-12T03:22:12.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:12 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:12.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:12 smithi033 ceph-mon[27552]: pgmap v109: 129 pgs: 10 creating+peering, 110 active+clean, 9 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-03-12T03:22:12.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:12 smithi033 ceph-mon[27552]: Deploying daemon prometheus.a on smithi033 2024-03-12T03:22:12.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:12 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:12.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:12 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:12.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:12 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:12.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:12 smithi007 ceph-mon[25800]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi033=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:12.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:12 smithi007 ceph-mon[25800]: Saving service grafana spec with placement smithi033=a;count:1 2024-03-12T03:22:12.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:12 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:12.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:12 smithi007 ceph-mon[25800]: pgmap v109: 129 pgs: 10 creating+peering, 110 active+clean, 9 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-03-12T03:22:12.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:12 smithi007 ceph-mon[25800]: Deploying daemon prometheus.a on smithi033 2024-03-12T03:22:12.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:12 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:12.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:12 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:12.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:12 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:12.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:12 smithi170 ceph-mon[27708]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi033=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:12.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:12 smithi170 ceph-mon[27708]: Saving service grafana spec with placement smithi033=a;count:1 2024-03-12T03:22:12.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:12 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:12.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:12 smithi170 ceph-mon[27708]: pgmap v109: 129 pgs: 10 creating+peering, 110 active+clean, 9 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-03-12T03:22:12.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:12 smithi170 ceph-mon[27708]: Deploying daemon prometheus.a on smithi033 2024-03-12T03:22:13.537 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:13 smithi007 ceph-mon[25800]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:22:13.537 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:13 smithi007 ceph-mon[25800]: Cluster is now healthy 2024-03-12T03:22:13.762 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:13 smithi170 ceph-mon[27708]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:22:13.762 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:13 smithi170 ceph-mon[27708]: Cluster is now healthy 2024-03-12T03:22:13.762 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:13 smithi033 ceph-mon[27552]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:22:13.762 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:13 smithi033 ceph-mon[27552]: Cluster is now healthy 2024-03-12T03:22:14.361 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-03-12T03:22:14.361 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBmyu9lkj1dFRAAo6ukyT24sjGfJF9m1C7rJg== 2024-03-12T03:22:14.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:14 smithi170 ceph-mon[27708]: pgmap v110: 129 pgs: 10 creating+peering, 119 active+clean; 579 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 45 KiB/s rd, 2.9 KiB/s wr, 97 op/s 2024-03-12T03:22:14.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.803 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:14 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/2590505802' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:22:14.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:14 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/2590505802' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:22:14.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.804 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:14 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:14 smithi007 ceph-mon[25800]: pgmap v110: 129 pgs: 10 creating+peering, 119 active+clean; 579 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 45 KiB/s rd, 2.9 KiB/s wr, 97 op/s 2024-03-12T03:22:14.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:14 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/2590505802' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:22:14.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:14 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/2590505802' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:22:14.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:14 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:14.952 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-12T03:22:14.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-12T03:22:14.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-12T03:22:14.995 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T03:22:15.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:14 smithi033 ceph-mon[27552]: pgmap v110: 129 pgs: 10 creating+peering, 119 active+clean; 579 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 45 KiB/s rd, 2.9 KiB/s wr, 97 op/s 2024-03-12T03:22:15.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:15.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:15.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:15.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:15.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:15.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:15.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:14 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/2590505802' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:22:15.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:14 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/2590505802' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:22:15.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:15.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:15.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:14 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:17.012 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:16 smithi033 ceph-mon[27552]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 112 KiB/s rd, 5.6 KiB/s wr, 249 op/s 2024-03-12T03:22:17.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:16 smithi007 ceph-mon[25800]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 112 KiB/s rd, 5.6 KiB/s wr, 249 op/s 2024-03-12T03:22:17.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:16 smithi170 ceph-mon[27708]: pgmap v111: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 112 KiB/s rd, 5.6 KiB/s wr, 249 op/s 2024-03-12T03:22:18.088 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T03:22:18.859 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:18 smithi033 ceph-mon[27552]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 90 KiB/s rd, 4.4 KiB/s wr, 200 op/s 2024-03-12T03:22:19.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:18 smithi007 ceph-mon[25800]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 90 KiB/s rd, 4.4 KiB/s wr, 200 op/s 2024-03-12T03:22:19.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:18 smithi170 ceph-mon[27708]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 90 KiB/s rd, 4.4 KiB/s wr, 200 op/s 2024-03-12T03:22:21.001 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:20 smithi033 ceph-mon[27552]: pgmap v113: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 76 KiB/s rd, 3.7 KiB/s wr, 169 op/s 2024-03-12T03:22:21.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:20 smithi007 ceph-mon[25800]: pgmap v113: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 76 KiB/s rd, 3.7 KiB/s wr, 169 op/s 2024-03-12T03:22:21.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:20 smithi170 ceph-mon[27708]: pgmap v113: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 76 KiB/s rd, 3.7 KiB/s wr, 169 op/s 2024-03-12T03:22:21.907 INFO:teuthology.orchestra.run.smithi033.stdout:[client.1] 2024-03-12T03:22:21.907 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBtyu9l3EvBNRAAzCA8NJquGsLKbT/8K+ytwQ== 2024-03-12T03:22:23.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:22 smithi033 ceph-mon[27552]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 68 KiB/s rd, 3.2 KiB/s wr, 152 op/s 2024-03-12T03:22:23.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:22 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:23.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:22 smithi033 ceph-mon[27552]: from='client.? 172.21.15.33:0/802503449' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:22:23.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:22 smithi033 ceph-mon[27552]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:22:23.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:22 smithi033 ceph-mon[27552]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:22:23.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:22 smithi007 ceph-mon[25800]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 68 KiB/s rd, 3.2 KiB/s wr, 152 op/s 2024-03-12T03:22:23.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:22 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:23.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:22 smithi007 ceph-mon[25800]: from='client.? 172.21.15.33:0/802503449' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:22:23.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:22 smithi007 ceph-mon[25800]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:22:23.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:22 smithi007 ceph-mon[25800]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:22:23.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:22 smithi170 ceph-mon[27708]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 68 KiB/s rd, 3.2 KiB/s wr, 152 op/s 2024-03-12T03:22:23.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:22 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:23.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:22 smithi170 ceph-mon[27708]: from='client.? 172.21.15.33:0/802503449' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:22:23.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:22 smithi170 ceph-mon[27708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:22:23.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:22 smithi170 ceph-mon[27708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:22:24.813 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:24 smithi033 ceph-mon[27552]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 60 KiB/s rd, 2.8 KiB/s wr, 132 op/s 2024-03-12T03:22:24.844 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-12T03:22:24.844 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-12T03:22:24.844 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-12T03:22:24.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:24 smithi007 ceph-mon[25800]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 60 KiB/s rd, 2.8 KiB/s wr, 132 op/s 2024-03-12T03:22:24.932 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T03:22:25.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:24 smithi170 ceph-mon[27708]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 60 KiB/s rd, 2.8 KiB/s wr, 132 op/s 2024-03-12T03:22:25.183 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T03:22:25.723 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 systemd[1]: Starting Ceph prometheus.a for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:22:25.978 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 podman[35664]: 2024-03-12 03:22:25.625519865 +0000 UTC m=+0.019269565 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-12T03:22:25.978 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 podman[35664]: 2024-03-12 03:22:25.746595368 +0000 UTC m=+0.140345050 container create b307e5aa8866dec1630f66293071bdaa661559ce24cf490d70f4929168f25c50 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:22:25.978 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 podman[35664]: 2024-03-12 03:22:25.948407083 +0000 UTC m=+0.342156764 container init b307e5aa8866dec1630f66293071bdaa661559ce24cf490d70f4929168f25c50 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:22:25.978 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 podman[35664]: 2024-03-12 03:22:25.953381741 +0000 UTC m=+0.347131423 container start b307e5aa8866dec1630f66293071bdaa661559ce24cf490d70f4929168f25c50 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:22:26.249 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.976Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-12T03:22:26.249 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.976Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-12T03:22:26.249 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.976Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi033 (none))" 2024-03-12T03:22:26.249 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.976Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-12T03:22:26.249 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.976Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-12T03:22:26.249 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.977Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-12T03:22:26.249 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.978Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-12T03:22:26.249 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.979Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-12T03:22:26.249 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.979Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-12T03:22:26.249 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.981Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-12T03:22:26.249 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.982Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.488µs 2024-03-12T03:22:26.250 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.982Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-12T03:22:26.250 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.982Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-12T03:22:26.250 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.982Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.847µs wal_replay_duration=291.714µs wbl_replay_duration=332ns total_replay_duration=346.501µs 2024-03-12T03:22:26.250 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.983Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-12T03:22:26.250 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.983Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-12T03:22:26.250 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:25 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:25.983Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-12T03:22:26.250 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:26 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:26.003Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.973526ms db_storage=691ns remote_storage=1.164µs web_handler=308ns query_engine=478ns scrape=443.593µs scrape_sd=86.052µs notify=1.015µs notify_sd=2.316µs rules=19.186902ms tracing=5.864µs 2024-03-12T03:22:26.250 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:26 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:26.003Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-12T03:22:26.250 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:26 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:22:26.003Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-12T03:22:26.250 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:26 smithi033 bash[35664]: b307e5aa8866dec1630f66293071bdaa661559ce24cf490d70f4929168f25c50 2024-03-12T03:22:26.250 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:22:26 smithi033 systemd[1]: Started Ceph prometheus.a for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:22:27.069 INFO:teuthology.orchestra.run.smithi170.stdout:[client.2] 2024-03-12T03:22:27.070 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBzyu9lRSWqAxAAZIrE82gjjdlzp3a5gp9fdw== 2024-03-12T03:22:27.347 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:27 smithi033 ceph-mon[27552]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 40 KiB/s rd, 1.8 KiB/s wr, 90 op/s 2024-03-12T03:22:27.347 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:27 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.348 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:27 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.348 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:27 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.348 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:27 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T03:22:27.348 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:27 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.348 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:27 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.348 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:27 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.348 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:27 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.348 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:27 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: ignoring --setuser ceph since I am not root 2024-03-12T03:22:27.348 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:27 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: ignoring --setgroup ceph since I am not root 2024-03-12T03:22:27.348 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:27 smithi033 ceph-mgr[29260]: -- 172.21.15.33:0/136735869 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a8a4df05a0 con 0x55a8a4dc4c00 2024-03-12T03:22:27.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:27 smithi007 ceph-mon[25800]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 40 KiB/s rd, 1.8 KiB/s wr, 90 op/s 2024-03-12T03:22:27.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:27 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:27 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:27 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:27 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T03:22:27.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:27 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:27 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:27 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:27 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:27 smithi170 ceph-mon[27708]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 40 KiB/s rd, 1.8 KiB/s wr, 90 op/s 2024-03-12T03:22:27.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:27 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:27 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:27 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:27 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T03:22:27.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:27 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.350 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:27 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.351 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:27 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.351 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:27 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' 2024-03-12T03:22:27.351 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:27 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: ignoring --setuser ceph since I am not root 2024-03-12T03:22:27.351 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:27 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: ignoring --setgroup ceph since I am not root 2024-03-12T03:22:27.351 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:27 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: ignoring --setuser ceph since I am not root 2024-03-12T03:22:27.351 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:27 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: ignoring --setgroup ceph since I am not root 2024-03-12T03:22:27.351 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:27 smithi170 ceph-mgr[30239]: -- 172.21.15.170:0/3620728909 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d779c8a5a0 con 0x55d779c5ec00 2024-03-12T03:22:27.696 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-12T03:22:27.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-12T03:22:27.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-12T03:22:27.740 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-12T03:22:27.741 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-12T03:22:27.741 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-12T03:22:27.767 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:27 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:27.707+0000 7f20a34f2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:22:27.990 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:22:28.012 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:27 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:27.668+0000 7f30ec121200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:22:28.013 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:27 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:27.763+0000 7f30ec121200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:22:28.064 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:27 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:27.804+0000 7f20a34f2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:22:28.065 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:27 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:27.696+0000 7f1110825200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:22:28.065 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:27 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:27.795+0000 7f1110825200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:22:28.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:28 smithi007 ceph-mon[25800]: from='client.? 172.21.15.170:0/3820467984' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:22:28.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:28 smithi007 ceph-mon[25800]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:22:28.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:28 smithi007 ceph-mon[25800]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:22:28.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:28 smithi007 ceph-mon[25800]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T03:22:28.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:28 smithi007 ceph-mon[25800]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-12T03:22:28.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:28 smithi170 ceph-mon[27708]: from='client.? 172.21.15.170:0/3820467984' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:22:28.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:28 smithi170 ceph-mon[27708]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:22:28.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:28 smithi170 ceph-mon[27708]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:22:28.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:28 smithi170 ceph-mon[27708]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T03:22:28.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:28 smithi170 ceph-mon[27708]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-12T03:22:28.404 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:28 smithi033 ceph-mon[27552]: from='client.? 172.21.15.170:0/3820467984' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:22:28.404 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:28 smithi033 ceph-mon[27552]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:22:28.405 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:28 smithi033 ceph-mon[27552]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:22:28.405 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:28 smithi033 ceph-mon[27552]: from='mgr.14150 172.21.15.7:0/3058717353' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T03:22:28.405 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:28 smithi033 ceph-mon[27552]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-12T03:22:28.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:28 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:28.427+0000 7f30ec121200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:22:28.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:28 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:28.525+0000 7f30ec121200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:22:28.676 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:28 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:28.673+0000 7f30ec121200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:22:28.727 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:28 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:28.487+0000 7f20a34f2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:22:28.727 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:28 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:28.582+0000 7f20a34f2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:22:28.727 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:28 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:28.725+0000 7f20a34f2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:22:28.774 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:28 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:28.445+0000 7f1110825200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:22:28.775 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:28 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:28.537+0000 7f1110825200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:22:28.775 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:28 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:28.677+0000 7f1110825200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:22:28.775 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:28 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:28.772+0000 7f1110825200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:22:29.013 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:28 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:28.772+0000 7f30ec121200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:22:29.013 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:28 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:28.858+0000 7f30ec121200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:22:29.098 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:28 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:28.823+0000 7f20a34f2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:22:29.098 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:28 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:28.907+0000 7f20a34f2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:22:29.113 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:28 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:28.854+0000 7f1110825200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:22:29.411 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:29 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:29.120+0000 7f1110825200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:22:29.411 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:29 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:29.228+0000 7f1110825200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:22:29.411 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:29 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:29.315+0000 7f1110825200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:22:29.412 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:29 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:29.409+0000 7f1110825200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:22:29.429 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:29 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:29.127+0000 7f30ec121200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:22:29.430 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:29 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:29.240+0000 7f30ec121200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:22:29.430 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:29 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:29.329+0000 7f30ec121200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:22:29.430 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:29 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:29.427+0000 7f30ec121200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:22:29.488 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:29 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:29.176+0000 7f20a34f2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:22:29.488 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:29 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:29.297+0000 7f20a34f2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:22:29.488 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:29 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:29.387+0000 7f20a34f2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:22:29.488 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:29 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:29.485+0000 7f20a34f2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:22:29.762 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:29 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:29.605+0000 7f30ec121200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:22:29.848 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:29 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:29.668+0000 7f20a34f2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:22:29.863 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:29 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:29.583+0000 7f1110825200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:22:29.993 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:22:30.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:30 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/297572480' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:22:30.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:30 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/297572480' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:22:30.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:30 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/297572480' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:22:30.672 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":431519931},{"type":"v1","addr":"172.21.15.7:6801","nonce":431519931}]},"active_addr":"172.21.15.7:6801/431519931","active_change":"2024-03-12T03:19:16.129275+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.7:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":45041351}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":2131816396}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":1515431171}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":1504240953}]}]} 2024-03-12T03:22:30.678 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-12T03:22:30.678 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-12T03:22:30.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:22:30.863 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:30 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:30.535+0000 7f1110825200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:22:30.863 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:30 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:30.665+0000 7f1110825200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:22:30.920 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:22:31.013 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:30 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:30.589+0000 7f30ec121200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:22:31.013 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:30 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:30.722+0000 7f30ec121200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:22:31.049 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:30 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:30.672+0000 7f20a34f2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:22:31.049 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:30 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:30.812+0000 7f20a34f2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:22:31.290 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:31 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:31.004+0000 7f1110825200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:22:31.290 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:31 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:31.093+0000 7f1110825200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:22:31.290 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:31 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:31.287+0000 7f1110825200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:22:31.304 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:31 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:31.172+0000 7f20a34f2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:22:31.304 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:31 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:31.267+0000 7f20a34f2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:22:31.354 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:31 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:31.059+0000 7f30ec121200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:22:31.354 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:31 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:31.152+0000 7f30ec121200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:22:31.598 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:31 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:31.472+0000 7f20a34f2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:22:31.613 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:31 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:31.471+0000 7f1110825200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:22:31.763 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:31 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:31.352+0000 7f30ec121200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:22:31.763 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:31 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:31.540+0000 7f30ec121200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:22:31.961 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:31 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:31.663+0000 7f20a34f2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:22:32.013 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:31 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:31.753+0000 7f1110825200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:22:32.013 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:32 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:32.011+0000 7f1110825200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:22:32.093 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:31 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:31.830+0000 7f30ec121200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:22:32.229 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:31 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:31.958+0000 7f20a34f2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:22:32.363 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:32 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:32.247+0000 7f1110825200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:22:32.480 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:32 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:32.227+0000 7f20a34f2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:22:32.512 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:32 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:32.091+0000 7f30ec121200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:22:32.513 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:32 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:32.333+0000 7f30ec121200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:22:32.843 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:22:32.843 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":30,"fsid":"fe67b1e4-e01e-11ee-95c8-87774f69a715","created":"2024-03-12T03:18:17.656051+0000","modified":"2024-03-12T03:22:11.209071+0000","last_up_change":"2024-03-12T03:21:47.536306+0000","last_in_change":"2024-03-12T03:21:26.563938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:21:48.477722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-12T03:22:03.130482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-12T03:22:04.512728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-12T03:22:06.464837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-12T03:22:08.514392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9444b255-7dd8-476c-a981-596313fa6856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":4033945070},{"type":"v1","addr":"172.21.15.7:6803","nonce":4033945070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":4033945070},{"type":"v1","addr":"172.21.15.7:6805","nonce":4033945070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":4033945070},{"type":"v1","addr":"172.21.15.7:6809","nonce":4033945070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":4033945070},{"type":"v1","addr":"172.21.15.7:6807","nonce":4033945070}]},"public_addr":"172.21.15.7:6803/4033945070","cluster_addr":"172.21.15.7:6805/4033945070","heartbeat_back_addr":"172.21.15.7:6809/4033945070","heartbeat_front_addr":"172.21.15.7:6807/4033945070","state":["exists","up"]},{"osd":1,"uuid":"06e71f2a-abc2-43f6-af2c-67eb8cba66f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":3621746125},{"type":"v1","addr":"172.21.15.33:6801","nonce":3621746125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":3621746125},{"type":"v1","addr":"172.21.15.33:6803","nonce":3621746125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":3621746125},{"type":"v1","addr":"172.21.15.33:6807","nonce":3621746125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":3621746125},{"type":"v1","addr":"172.21.15.33:6805","nonce":3621746125}]},"public_addr":"172.21.15.33:6801/3621746125","cluster_addr":"172.21.15.33:6803/3621746125","heartbeat_back_addr":"172.21.15.33:6807/3621746125","heartbeat_front_addr":"172.21.15.33:6805/3621746125","state":["exists","up"]},{"osd":2,"uuid":"8533b7d3-a661-4cc0-ad46-e7ab0faaa107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3615959269},{"type":"v1","addr":"172.21.15.170:6801","nonce":3615959269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3615959269},{"type":"v1","addr":"172.21.15.170:6803","nonce":3615959269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3615959269},{"type":"v1","addr":"172.21.15.170:6807","nonce":3615959269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3615959269},{"type":"v1","addr":"172.21.15.170:6805","nonce":3615959269}]},"public_addr":"172.21.15.170:6801/3615959269","cluster_addr":"172.21.15.170:6803/3615959269","heartbeat_back_addr":"172.21.15.170:6807/3615959269","heartbeat_front_addr":"172.21.15.170:6805/3615959269","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:20:51.583068+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:21:17.997089+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:21:45.317595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/821359088":"2024-03-13T03:19:16.129168+0000","172.21.15.7:0/3516588619":"2024-03-13T03:19:16.129168+0000","172.21.15.7:6800/1100758856":"2024-03-13T03:18:49.567015+0000","172.21.15.7:0/3508574268":"2024-03-13T03:19:16.129168+0000","172.21.15.7:6800/928905743":"2024-03-13T03:19:16.129168+0000","172.21.15.7:0/3944002862":"2024-03-13T03:18:49.567015+0000","172.21.15.7:6801/928905743":"2024-03-13T03:19:16.129168+0000","172.21.15.7:6801/1100758856":"2024-03-13T03:18:49.567015+0000","172.21.15.7:0/2116937091":"2024-03-13T03:18:49.567015+0000","172.21.15.7:0/503027408":"2024-03-13T03:18:49.567015+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:22:32.843 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:32 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:32.478+0000 7f20a34f2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:22:33.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:32 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/2688503927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:22:33.160 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:32 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/2688503927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:22:33.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:32 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/2688503927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:22:33.363 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:33 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:33.040+0000 7f1110825200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:22:33.512 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:33 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:33.157+0000 7f30ec121200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:22:33.573 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:33 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:33.321+0000 7f20a34f2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:22:33.600 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-12T03:22:33.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:22:33.748 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:33 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:33.635+0000 7f1110825200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:22:33.844 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:22:34.012 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:33 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:33.739+0000 7f30ec121200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:22:34.013 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:33 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:33.840+0000 7f30ec121200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:22:34.028 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:33 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:33.744+0000 7f1110825200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:22:34.028 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:33 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:33.962+0000 7f1110825200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:22:34.028 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:33 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: [12/Mar/2024:03:22:33] ENGINE Bus STARTING 2024-03-12T03:22:34.210 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:33 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:33.923+0000 7f20a34f2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:22:34.210 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:34 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:34.028+0000 7f20a34f2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:22:34.210 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:34 smithi007 ceph-mon[25800]: Standby manager daemon z restarted 2024-03-12T03:22:34.210 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:34 smithi007 ceph-mon[25800]: Standby manager daemon z started 2024-03-12T03:22:34.210 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:34 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.170:0/3385677387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T03:22:34.210 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:34 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.170:0/3385677387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:22:34.210 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:34 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.170:0/3385677387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T03:22:34.210 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:34 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.170:0/3385677387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:22:34.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:34 smithi170 ceph-mon[27708]: Standby manager daemon z restarted 2024-03-12T03:22:34.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:34 smithi170 ceph-mon[27708]: Standby manager daemon z started 2024-03-12T03:22:34.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:34 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.170:0/3385677387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T03:22:34.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:34 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.170:0/3385677387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:22:34.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:34 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.170:0/3385677387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T03:22:34.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:34 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.170:0/3385677387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:22:34.364 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:34 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: CherryPy Checker: 2024-03-12T03:22:34.364 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:34 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: The Application mounted at '' has an empty config. 2024-03-12T03:22:34.364 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:34 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: 2024-03-12T03:22:34.364 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:34 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: [12/Mar/2024:03:22:34] ENGINE Serving on http://:::9283 2024-03-12T03:22:34.364 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:22:34 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z[30216]: [12/Mar/2024:03:22:34] ENGINE Bus STARTED 2024-03-12T03:22:34.453 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:34 smithi033 ceph-mon[27552]: Standby manager daemon z restarted 2024-03-12T03:22:34.453 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:34 smithi033 ceph-mon[27552]: Standby manager daemon z started 2024-03-12T03:22:34.453 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:34 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.170:0/3385677387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T03:22:34.453 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:34 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.170:0/3385677387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:22:34.453 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:34 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.170:0/3385677387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T03:22:34.453 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:34 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.170:0/3385677387' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:22:34.454 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:34 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:34.061+0000 7f30ec121200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:22:34.454 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:34 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: [12/Mar/2024:03:22:34] ENGINE Bus STARTING 2024-03-12T03:22:34.454 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:34 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: CherryPy Checker: 2024-03-12T03:22:34.454 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:34 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: The Application mounted at '' has an empty config. 2024-03-12T03:22:34.454 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:34 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: 2024-03-12T03:22:34.454 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:34 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: [12/Mar/2024:03:22:34] ENGINE Serving on http://:::9283 2024-03-12T03:22:34.454 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:22:34 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x[29237]: [12/Mar/2024:03:22:34] ENGINE Bus STARTED 2024-03-12T03:22:34.508 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:34 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:34.262+0000 7f20a34f2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:22:34.508 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:34 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: [12/Mar/2024:03:22:34] ENGINE Bus STARTING 2024-03-12T03:22:34.848 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:34 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: CherryPy Checker: 2024-03-12T03:22:34.849 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:34 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: The Application mounted at '' has an empty config. 2024-03-12T03:22:34.849 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:34 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:22:34.849 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:34 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: [12/Mar/2024:03:22:34] ENGINE Serving on http://:::9283 2024-03-12T03:22:34.849 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:34 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: [12/Mar/2024:03:22:34] ENGINE Bus STARTED 2024-03-12T03:22:35.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-12T03:22:35.030 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: Standby manager daemon x restarted 2024-03-12T03:22:35.284 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-12T03:22:35.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: Standby manager daemon x restarted 2024-03-12T03:22:35.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: Standby manager daemon x started 2024-03-12T03:22:35.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.33:0/1756063790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:22:35.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.33:0/1756063790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:22:35.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.33:0/1756063790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:22:35.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.? 172.21.15.33:0/1756063790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:22:35.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: Active manager daemon y restarted 2024-03-12T03:22:35.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: Activating manager daemon y 2024-03-12T03:22:35.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: osdmap e31: 3 total, 3 up, 3 in 2024-03-12T03:22:35.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: mgrmap e19: y(active, starting, since 0.0112411s), standbys: x, z 2024-03-12T03:22:35.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:22:35.285 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:22:35.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:22:35.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:22:35.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:22:35.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T03:22:35.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:22:35.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:22:35.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:22:35.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:22:35.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:22:35.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:22:35.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: Manager daemon y is now available 2024-03-12T03:22:35.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:35.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:35.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:35.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:22:35.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:22:35.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]': finished 2024-03-12T03:22:35.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:22:35.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]': finished 2024-03-12T03:22:35.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]': finished 2024-03-12T03:22:35.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:35.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:35.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:35.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:35.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:22:35.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:22:35.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: Standby manager daemon x started 2024-03-12T03:22:35.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.33:0/1756063790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:22:35.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.33:0/1756063790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:22:35.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.33:0/1756063790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:22:35.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.? 172.21.15.33:0/1756063790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:22:35.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: Active manager daemon y restarted 2024-03-12T03:22:35.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: Activating manager daemon y 2024-03-12T03:22:35.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: osdmap e31: 3 total, 3 up, 3 in 2024-03-12T03:22:35.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: mgrmap e19: y(active, starting, since 0.0112411s), standbys: x, z 2024-03-12T03:22:35.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:22:35.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:22:35.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:22:35.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:22:35.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:22:35.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T03:22:35.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:22:35.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:22:35.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:22:35.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:22:35.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:22:35.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:22:35.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: Manager daemon y is now available 2024-03-12T03:22:35.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:35.365 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:35.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:35.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:22:35.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:22:35.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]': finished 2024-03-12T03:22:35.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:22:35.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]': finished 2024-03-12T03:22:35.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]': finished 2024-03-12T03:22:35.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:35.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:35.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:35.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:35.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:22:35.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:22:35.370 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-12T03:22:35.370 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: Standby manager daemon x restarted 2024-03-12T03:22:35.370 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: Standby manager daemon x started 2024-03-12T03:22:35.371 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.33:0/1756063790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:22:35.371 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.33:0/1756063790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:22:35.371 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.33:0/1756063790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:22:35.371 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.? 172.21.15.33:0/1756063790' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:22:35.371 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: Active manager daemon y restarted 2024-03-12T03:22:35.371 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: Activating manager daemon y 2024-03-12T03:22:35.371 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: osdmap e31: 3 total, 3 up, 3 in 2024-03-12T03:22:35.371 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: mgrmap e19: y(active, starting, since 0.0112411s), standbys: x, z 2024-03-12T03:22:35.371 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:22:35.371 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:22:35.371 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:22:35.371 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:22:35.371 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:22:35.372 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T03:22:35.372 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:22:35.372 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:22:35.372 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:22:35.372 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:22:35.372 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:22:35.372 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:22:35.372 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: Manager daemon y is now available 2024-03-12T03:22:35.372 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:35.372 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:35.372 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:35.372 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:22:35.372 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:22:35.372 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]': finished 2024-03-12T03:22:35.372 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:22:35.373 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]': finished 2024-03-12T03:22:35.373 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]': finished 2024-03-12T03:22:35.373 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:35.373 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:35.373 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:35.373 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:35.373 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:22:35.373 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:22:35.896 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:22:35.896 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":31,"fsid":"fe67b1e4-e01e-11ee-95c8-87774f69a715","created":"2024-03-12T03:18:17.656051+0000","modified":"2024-03-12T03:22:34.277128+0000","last_up_change":"2024-03-12T03:21:47.536306+0000","last_in_change":"2024-03-12T03:21:26.563938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:21:48.477722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-12T03:22:03.130482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-12T03:22:04.512728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-12T03:22:06.464837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-12T03:22:08.514392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9444b255-7dd8-476c-a981-596313fa6856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":4033945070},{"type":"v1","addr":"172.21.15.7:6803","nonce":4033945070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":4033945070},{"type":"v1","addr":"172.21.15.7:6805","nonce":4033945070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":4033945070},{"type":"v1","addr":"172.21.15.7:6809","nonce":4033945070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":4033945070},{"type":"v1","addr":"172.21.15.7:6807","nonce":4033945070}]},"public_addr":"172.21.15.7:6803/4033945070","cluster_addr":"172.21.15.7:6805/4033945070","heartbeat_back_addr":"172.21.15.7:6809/4033945070","heartbeat_front_addr":"172.21.15.7:6807/4033945070","state":["exists","up"]},{"osd":1,"uuid":"06e71f2a-abc2-43f6-af2c-67eb8cba66f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":3621746125},{"type":"v1","addr":"172.21.15.33:6801","nonce":3621746125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":3621746125},{"type":"v1","addr":"172.21.15.33:6803","nonce":3621746125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":3621746125},{"type":"v1","addr":"172.21.15.33:6807","nonce":3621746125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":3621746125},{"type":"v1","addr":"172.21.15.33:6805","nonce":3621746125}]},"public_addr":"172.21.15.33:6801/3621746125","cluster_addr":"172.21.15.33:6803/3621746125","heartbeat_back_addr":"172.21.15.33:6807/3621746125","heartbeat_front_addr":"172.21.15.33:6805/3621746125","state":["exists","up"]},{"osd":2,"uuid":"8533b7d3-a661-4cc0-ad46-e7ab0faaa107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3615959269},{"type":"v1","addr":"172.21.15.170:6801","nonce":3615959269}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3615959269},{"type":"v1","addr":"172.21.15.170:6803","nonce":3615959269}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3615959269},{"type":"v1","addr":"172.21.15.170:6807","nonce":3615959269}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3615959269},{"type":"v1","addr":"172.21.15.170:6805","nonce":3615959269}]},"public_addr":"172.21.15.170:6801/3615959269","cluster_addr":"172.21.15.170:6803/3615959269","heartbeat_back_addr":"172.21.15.170:6807/3615959269","heartbeat_front_addr":"172.21.15.170:6805/3615959269","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:20:51.583068+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:21:17.997089+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:21:45.317595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/2131816396":"2024-03-13T03:22:34.277096+0000","172.21.15.7:6801/431519931":"2024-03-13T03:22:34.277096+0000","172.21.15.7:6800/431519931":"2024-03-13T03:22:34.277096+0000","172.21.15.7:0/821359088":"2024-03-13T03:19:16.129168+0000","172.21.15.7:0/3516588619":"2024-03-13T03:19:16.129168+0000","172.21.15.7:6800/1100758856":"2024-03-13T03:18:49.567015+0000","172.21.15.7:0/3508574268":"2024-03-13T03:19:16.129168+0000","172.21.15.7:6800/928905743":"2024-03-13T03:19:16.129168+0000","172.21.15.7:0/45041351":"2024-03-13T03:22:34.277096+0000","172.21.15.7:0/3944002862":"2024-03-13T03:18:49.567015+0000","172.21.15.7:6801/928905743":"2024-03-13T03:19:16.129168+0000","172.21.15.7:6801/1100758856":"2024-03-13T03:18:49.567015+0000","172.21.15.7:0/1504240953":"2024-03-13T03:22:34.277096+0000","172.21.15.7:0/2116937091":"2024-03-13T03:18:49.567015+0000","172.21.15.7:0/1515431171":"2024-03-13T03:22:34.277096+0000","172.21.15.7:0/503027408":"2024-03-13T03:18:49.567015+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:22:36.098 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:35 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: ::ffff:172.21.15.33 - - [12/Mar/2024:03:22:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:22:36.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: Updating smithi007:/etc/ceph/ceph.conf 2024-03-12T03:22:36.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-12T03:22:36.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-12T03:22:36.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: [12/Mar/2024:03:22:34] ENGINE Bus STARTING 2024-03-12T03:22:36.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: [12/Mar/2024:03:22:34] ENGINE Serving on https://172.21.15.7:7151 2024-03-12T03:22:36.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: Updating smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:22:36.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: [12/Mar/2024:03:22:34] ENGINE Serving on http://172.21.15.7:8765 2024-03-12T03:22:36.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: [12/Mar/2024:03:22:34] ENGINE Bus STARTED 2024-03-12T03:22:36.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: Updating smithi007:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:22:36.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: Updating smithi033:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:22:36.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:22:36.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: mgrmap e20: y(active, since 1.01689s), standbys: x, z 2024-03-12T03:22:36.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:36 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/4171922979' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:22:36.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: Updating smithi007:/etc/ceph/ceph.conf 2024-03-12T03:22:36.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-12T03:22:36.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-12T03:22:36.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: [12/Mar/2024:03:22:34] ENGINE Bus STARTING 2024-03-12T03:22:36.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: [12/Mar/2024:03:22:34] ENGINE Serving on https://172.21.15.7:7151 2024-03-12T03:22:36.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: Updating smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:22:36.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: [12/Mar/2024:03:22:34] ENGINE Serving on http://172.21.15.7:8765 2024-03-12T03:22:36.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: [12/Mar/2024:03:22:34] ENGINE Bus STARTED 2024-03-12T03:22:36.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: Updating smithi007:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:22:36.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: Updating smithi033:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:22:36.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:22:36.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: mgrmap e20: y(active, since 1.01689s), standbys: x, z 2024-03-12T03:22:36.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.114 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:36 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/4171922979' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:22:36.314 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: Updating smithi007:/etc/ceph/ceph.conf 2024-03-12T03:22:36.314 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-12T03:22:36.314 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-12T03:22:36.314 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: [12/Mar/2024:03:22:34] ENGINE Bus STARTING 2024-03-12T03:22:36.314 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: [12/Mar/2024:03:22:34] ENGINE Serving on https://172.21.15.7:7151 2024-03-12T03:22:36.314 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: Updating smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:22:36.314 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: [12/Mar/2024:03:22:34] ENGINE Serving on http://172.21.15.7:8765 2024-03-12T03:22:36.314 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: [12/Mar/2024:03:22:34] ENGINE Bus STARTED 2024-03-12T03:22:36.314 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: Updating smithi007:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:22:36.314 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: Updating smithi033:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:22:36.315 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:22:36.315 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: mgrmap e20: y(active, since 1.01689s), standbys: x, z 2024-03-12T03:22:36.315 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.315 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.315 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.315 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.315 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.315 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.315 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.315 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:36.315 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:36 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/4171922979' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:22:36.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-12T03:22:36.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-12T03:22:36.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-12T03:22:36.793 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:22:36.800 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:22:36.808 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:22:37.166 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:37 smithi007 ceph-mon[25800]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:22:37.166 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:37 smithi007 ceph-mon[25800]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:22:37.166 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:37 smithi007 ceph-mon[25800]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:37.166 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:37 smithi007 ceph-mon[25800]: Updating smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:22:37.166 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:37 smithi007 ceph-mon[25800]: Updating smithi007:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:22:37.166 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:37 smithi007 ceph-mon[25800]: Updating smithi033:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:22:37.166 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:37 smithi007 ceph-mon[25800]: Deploying daemon alertmanager.a on smithi007 2024-03-12T03:22:37.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:37 smithi170 ceph-mon[27708]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:22:37.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:37 smithi170 ceph-mon[27708]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:22:37.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:37 smithi170 ceph-mon[27708]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:37.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:37 smithi170 ceph-mon[27708]: Updating smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:22:37.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:37 smithi170 ceph-mon[27708]: Updating smithi007:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:22:37.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:37 smithi170 ceph-mon[27708]: Updating smithi033:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:22:37.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:37 smithi170 ceph-mon[27708]: Deploying daemon alertmanager.a on smithi007 2024-03-12T03:22:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:37 smithi033 ceph-mon[27552]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:22:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:37 smithi033 ceph-mon[27552]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:22:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:37 smithi033 ceph-mon[27552]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:37 smithi033 ceph-mon[27552]: Updating smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:22:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:37 smithi033 ceph-mon[27552]: Updating smithi007:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:22:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:37 smithi033 ceph-mon[27552]: Updating smithi033:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:22:37.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:37 smithi033 ceph-mon[27552]: Deploying daemon alertmanager.a on smithi007 2024-03-12T03:22:38.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:38 smithi007 ceph-mon[25800]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:38.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:38 smithi007 ceph-mon[25800]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-12T03:22:38.353 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:38 smithi170 ceph-mon[27708]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:38.353 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:38 smithi170 ceph-mon[27708]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-12T03:22:38.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:38 smithi033 ceph-mon[27552]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:38.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:38 smithi033 ceph-mon[27552]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-12T03:22:40.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:40 smithi007 ceph-mon[25800]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:40.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:40 smithi007 ceph-mon[25800]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-12T03:22:40.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:40 smithi170 ceph-mon[27708]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:40.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:40 smithi170 ceph-mon[27708]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-12T03:22:40.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:40 smithi033 ceph-mon[27552]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:40.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:40 smithi033 ceph-mon[27552]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-12T03:22:42.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:42 smithi007 ceph-mon[25800]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:42.262 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:42 smithi033 ceph-mon[27552]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:42.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:42 smithi170 ceph-mon[27708]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:43.495 INFO:teuthology.orchestra.run.smithi007.stdout:34359738392 2024-03-12T03:22:43.496 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-12T03:22:43.910 INFO:teuthology.orchestra.run.smithi007.stdout:55834574867 2024-03-12T03:22:43.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-12T03:22:44.303 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:44 smithi033 ceph-mon[27552]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-12T03:22:44.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:44 smithi007 ceph-mon[25800]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-12T03:22:44.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:44 smithi170 ceph-mon[27708]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-12T03:22:46.048 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:45 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: ::ffff:172.21.15.33 - - [12/Mar/2024:03:22:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:22:46.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:46 smithi007 ceph-mon[25800]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-12T03:22:46.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:46 smithi170 ceph-mon[27708]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-12T03:22:46.460 INFO:teuthology.orchestra.run.smithi007.stdout:77309411341 2024-03-12T03:22:46.460 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-12T03:22:46.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:46 smithi033 ceph-mon[27552]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-12T03:22:48.286 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:22:48.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:48 smithi007 ceph-mon[25800]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:22:48.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:48 smithi170 ceph-mon[27708]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:22:48.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:48 smithi033 ceph-mon[27552]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:22:48.730 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:22:48.779 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:22:50.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:50 smithi007 ceph-mon[25800]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T03:22:50.099 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:50 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:50.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:50 smithi170 ceph-mon[27708]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T03:22:50.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:50 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:50.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:50 smithi033 ceph-mon[27552]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T03:22:50.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:50 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:52.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:52 smithi007 ceph-mon[25800]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T03:22:52.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:52 smithi170 ceph-mon[27708]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T03:22:52.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:52 smithi033 ceph-mon[27552]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T03:22:52.744 INFO:teuthology.orchestra.run.smithi007.stdout:55834574868 2024-03-12T03:22:52.821 INFO:teuthology.orchestra.run.smithi007.stdout:34359738394 2024-03-12T03:22:53.322 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:53 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/4050201724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:22:53.322 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:53 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/3034364137' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:22:53.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:53 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/4050201724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:22:53.366 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:53 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/3034364137' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:22:53.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:53 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/4050201724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:22:53.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:53 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/3034364137' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:22:53.702 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574867 got 55834574868 for osd.1 2024-03-12T03:22:53.703 DEBUG:teuthology.parallel:result is None 2024-03-12T03:22:54.139 INFO:teuthology.orchestra.run.smithi007.stdout:77309411343 2024-03-12T03:22:54.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:54 smithi007 ceph-mon[25800]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T03:22:54.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:54 smithi170 ceph-mon[27708]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T03:22:54.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:54 smithi033 ceph-mon[27552]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T03:22:54.598 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738392 got 34359738394 for osd.0 2024-03-12T03:22:54.598 DEBUG:teuthology.parallel:result is None 2024-03-12T03:22:55.098 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:55 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/686439341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:22:55.340 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:55 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/686439341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:22:55.376 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411341 got 77309411343 for osd.2 2024-03-12T03:22:55.376 DEBUG:teuthology.parallel:result is None 2024-03-12T03:22:55.376 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-12T03:22:55.377 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:22:55.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:55 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/686439341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:22:55.598 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:22:55 smithi007 podman[48126]: 2024-03-12 03:22:55.464766717 +0000 UTC m=+2.869169445 container create 47e2fba1cdb1c13d05389a7442d89893f6b33a1c097cbfa9d6072bcaaa25ca19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:22:55.887 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:22:56.012 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:55 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: ::ffff:172.21.15.33 - - [12/Mar/2024:03:22:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:22:56.012 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:22:55 smithi007 podman[48126]: 2024-03-12 03:22:55.725665723 +0000 UTC m=+3.130068398 container init 47e2fba1cdb1c13d05389a7442d89893f6b33a1c097cbfa9d6072bcaaa25ca19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:22:56.012 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:22:55 smithi007 podman[48126]: 2024-03-12 03:22:55.73077139 +0000 UTC m=+3.135174063 container start 47e2fba1cdb1c13d05389a7442d89893f6b33a1c097cbfa9d6072bcaaa25ca19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:22:56.012 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:22:55 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a[48326]: ts=2024-03-12T03:22:55.742Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-12T03:22:56.012 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:22:55 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a[48326]: ts=2024-03-12T03:22:55.742Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-12T03:22:56.013 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:22:55 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a[48326]: ts=2024-03-12T03:22:55.744Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.7 port=9094 2024-03-12T03:22:56.013 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:22:55 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a[48326]: ts=2024-03-12T03:22:55.745Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-12T03:22:56.013 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:22:55 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a[48326]: ts=2024-03-12T03:22:55.767Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:22:56.013 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:22:55 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a[48326]: ts=2024-03-12T03:22:55.768Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:22:56.013 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:22:55 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a[48326]: ts=2024-03-12T03:22:55.770Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-12T03:22:56.013 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:22:55 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a[48326]: ts=2024-03-12T03:22:55.770Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-12T03:22:56.013 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:22:55 smithi007 bash[48126]: 47e2fba1cdb1c13d05389a7442d89893f6b33a1c097cbfa9d6072bcaaa25ca19 2024-03-12T03:22:56.013 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:22:55 smithi007 systemd[1]: Started Ceph alertmanager.a for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:22:56.280 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:56 smithi007 ceph-mon[25800]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:56.280 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:56 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.280 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:56 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.280 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:56 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.280 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:56 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.281 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:56 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.281 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:56 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.281 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:56 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:22:56.281 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:56 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.281 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:56 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: [12/Mar/2024:03:22:56] ENGINE Bus STOPPING 2024-03-12T03:22:56.281 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:56 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: [12/Mar/2024:03:22:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:22:56.281 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:56 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: [12/Mar/2024:03:22:56] ENGINE Bus STOPPED 2024-03-12T03:22:56.281 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:56 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: [12/Mar/2024:03:22:56] ENGINE Bus STARTING 2024-03-12T03:22:56.281 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:56 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: [12/Mar/2024:03:22:56] ENGINE Serving on http://:::9283 2024-03-12T03:22:56.281 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:22:56 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: [12/Mar/2024:03:22:56] ENGINE Bus STARTED 2024-03-12T03:22:56.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:56 smithi170 ceph-mon[27708]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:56.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:56 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:56 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:56 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:56 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:56 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:56 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:56 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:22:56.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:56 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:56 smithi033 ceph-mon[27552]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:56.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:56 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:56 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:56 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:56 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:56 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:56 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:56.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:56 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:22:56.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:56 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:22:57.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:57 smithi007 ceph-mon[25800]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T03:22:57.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:57 smithi007 ceph-mon[25800]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:22:57.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:57 smithi007 ceph-mon[25800]: Deploying daemon grafana.a on smithi033 2024-03-12T03:22:57.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:57 smithi170 ceph-mon[27708]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T03:22:57.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:57 smithi170 ceph-mon[27708]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:22:57.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:57 smithi170 ceph-mon[27708]: Deploying daemon grafana.a on smithi033 2024-03-12T03:22:57.425 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:57 smithi033 ceph-mon[27552]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T03:22:57.425 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:57 smithi033 ceph-mon[27552]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:22:57.425 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:57 smithi033 ceph-mon[27552]: Deploying daemon grafana.a on smithi033 2024-03-12T03:22:57.848 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:22:57 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a[48326]: ts=2024-03-12T03:22:57.745Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00073311s 2024-03-12T03:22:58.273 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:22:58 smithi007 ceph-mon[25800]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:58.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:22:58 smithi170 ceph-mon[27708]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:58.434 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:22:58 smithi033 ceph-mon[27552]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:22:59.416 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:22:59.419 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-12T03:23:00.084 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-03-12T03:22:58.297888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595984,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":104932,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":101819,"kb_avail":281118236,"statfs":{"total":287972524032,"available":287865073664,"internally_reserved":0,"allocated":3158016,"data_stored":2100573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":104262992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002919"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283798+0000","last_change":"2024-03-12T03:22:10.224487+0000","last_active":"2024-03-12T03:22:34.283798+0000","last_peered":"2024-03-12T03:22:34.283798+0000","last_clean":"2024-03-12T03:22:34.283798+0000","last_became_active":"2024-03-12T03:22:10.223135+0000","last_became_peered":"2024-03-12T03:22:10.223135+0000","last_unstale":"2024-03-12T03:22:34.283798+0000","last_undegraded":"2024-03-12T03:22:34.283798+0000","last_fullsized":"2024-03-12T03:22:34.283798+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:23:15.142101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282965+0000","last_change":"2024-03-12T03:22:08.208026+0000","last_active":"2024-03-12T03:22:34.282965+0000","last_peered":"2024-03-12T03:22:34.282965+0000","last_clean":"2024-03-12T03:22:34.282965+0000","last_became_active":"2024-03-12T03:22:08.207641+0000","last_became_peered":"2024-03-12T03:22:08.207641+0000","last_unstale":"2024-03-12T03:22:34.282965+0000","last_undegraded":"2024-03-12T03:22:34.282965+0000","last_fullsized":"2024-03-12T03:22:34.282965+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:53:20.180584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285437+0000","last_change":"2024-03-12T03:22:04.194608+0000","last_active":"2024-03-12T03:22:34.285437+0000","last_peered":"2024-03-12T03:22:34.285437+0000","last_clean":"2024-03-12T03:22:34.285437+0000","last_became_active":"2024-03-12T03:22:04.193837+0000","last_became_peered":"2024-03-12T03:22:04.193837+0000","last_unstale":"2024-03-12T03:22:34.285437+0000","last_undegraded":"2024-03-12T03:22:34.285437+0000","last_fullsized":"2024-03-12T03:22:34.285437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:00:11.180424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284008+0000","last_change":"2024-03-12T03:22:06.200644+0000","last_active":"2024-03-12T03:22:34.284008+0000","last_peered":"2024-03-12T03:22:34.284008+0000","last_clean":"2024-03-12T03:22:34.284008+0000","last_became_active":"2024-03-12T03:22:06.199826+0000","last_became_peered":"2024-03-12T03:22:06.199826+0000","last_unstale":"2024-03-12T03:22:34.284008+0000","last_undegraded":"2024-03-12T03:22:34.284008+0000","last_fullsized":"2024-03-12T03:22:34.284008+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:09:11.736828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282489+0000","last_change":"2024-03-12T03:22:10.227009+0000","last_active":"2024-03-12T03:22:34.282489+0000","last_peered":"2024-03-12T03:22:34.282489+0000","last_clean":"2024-03-12T03:22:34.282489+0000","last_became_active":"2024-03-12T03:22:10.226585+0000","last_became_peered":"2024-03-12T03:22:10.226585+0000","last_unstale":"2024-03-12T03:22:34.282489+0000","last_undegraded":"2024-03-12T03:22:34.282489+0000","last_fullsized":"2024-03-12T03:22:34.282489+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:05:01.259966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283510+0000","last_change":"2024-03-12T03:22:08.213121+0000","last_active":"2024-03-12T03:22:34.283510+0000","last_peered":"2024-03-12T03:22:34.283510+0000","last_clean":"2024-03-12T03:22:34.283510+0000","last_became_active":"2024-03-12T03:22:08.212605+0000","last_became_peered":"2024-03-12T03:22:08.212605+0000","last_unstale":"2024-03-12T03:22:34.283510+0000","last_undegraded":"2024-03-12T03:22:34.283510+0000","last_fullsized":"2024-03-12T03:22:34.283510+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:16:18.411495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284265+0000","last_change":"2024-03-12T03:22:04.195506+0000","last_active":"2024-03-12T03:22:34.284265+0000","last_peered":"2024-03-12T03:22:34.284265+0000","last_clean":"2024-03-12T03:22:34.284265+0000","last_became_active":"2024-03-12T03:22:04.195068+0000","last_became_peered":"2024-03-12T03:22:04.195068+0000","last_unstale":"2024-03-12T03:22:34.284265+0000","last_undegraded":"2024-03-12T03:22:34.284265+0000","last_fullsized":"2024-03-12T03:22:34.284265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:38:18.952407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284924+0000","last_change":"2024-03-12T03:22:06.200042+0000","last_active":"2024-03-12T03:22:34.284924+0000","last_peered":"2024-03-12T03:22:34.284924+0000","last_clean":"2024-03-12T03:22:34.284924+0000","last_became_active":"2024-03-12T03:22:06.199705+0000","last_became_peered":"2024-03-12T03:22:06.199705+0000","last_unstale":"2024-03-12T03:22:34.284924+0000","last_undegraded":"2024-03-12T03:22:34.284924+0000","last_fullsized":"2024-03-12T03:22:34.284924+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:09:04.186783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282838+0000","last_change":"2024-03-12T03:22:10.218678+0000","last_active":"2024-03-12T03:22:34.282838+0000","last_peered":"2024-03-12T03:22:34.282838+0000","last_clean":"2024-03-12T03:22:34.282838+0000","last_became_active":"2024-03-12T03:22:10.218375+0000","last_became_peered":"2024-03-12T03:22:10.218375+0000","last_unstale":"2024-03-12T03:22:34.282838+0000","last_undegraded":"2024-03-12T03:22:34.282838+0000","last_fullsized":"2024-03-12T03:22:34.282838+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:07:46.506793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283805+0000","last_change":"2024-03-12T03:22:08.212813+0000","last_active":"2024-03-12T03:22:34.283805+0000","last_peered":"2024-03-12T03:22:34.283805+0000","last_clean":"2024-03-12T03:22:34.283805+0000","last_became_active":"2024-03-12T03:22:08.212249+0000","last_became_peered":"2024-03-12T03:22:08.212249+0000","last_unstale":"2024-03-12T03:22:34.283805+0000","last_undegraded":"2024-03-12T03:22:34.283805+0000","last_fullsized":"2024-03-12T03:22:34.283805+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:25:59.775838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285949+0000","last_change":"2024-03-12T03:22:04.201207+0000","last_active":"2024-03-12T03:22:34.285949+0000","last_peered":"2024-03-12T03:22:34.285949+0000","last_clean":"2024-03-12T03:22:34.285949+0000","last_became_active":"2024-03-12T03:22:04.200522+0000","last_became_peered":"2024-03-12T03:22:04.200522+0000","last_unstale":"2024-03-12T03:22:34.285949+0000","last_undegraded":"2024-03-12T03:22:34.285949+0000","last_fullsized":"2024-03-12T03:22:34.285949+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:21.224226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285839+0000","last_change":"2024-03-12T03:22:06.201984+0000","last_active":"2024-03-12T03:22:34.285839+0000","last_peered":"2024-03-12T03:22:34.285839+0000","last_clean":"2024-03-12T03:22:34.285839+0000","last_became_active":"2024-03-12T03:22:06.200775+0000","last_became_peered":"2024-03-12T03:22:06.200775+0000","last_unstale":"2024-03-12T03:22:34.285839+0000","last_undegraded":"2024-03-12T03:22:34.285839+0000","last_fullsized":"2024-03-12T03:22:34.285839+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:25:56.794935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283555+0000","last_change":"2024-03-12T03:22:10.220543+0000","last_active":"2024-03-12T03:22:34.283555+0000","last_peered":"2024-03-12T03:22:34.283555+0000","last_clean":"2024-03-12T03:22:34.283555+0000","last_became_active":"2024-03-12T03:22:10.219924+0000","last_became_peered":"2024-03-12T03:22:10.219924+0000","last_unstale":"2024-03-12T03:22:34.283555+0000","last_undegraded":"2024-03-12T03:22:34.283555+0000","last_fullsized":"2024-03-12T03:22:34.283555+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:49:30.900784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283358+0000","last_change":"2024-03-12T03:22:08.211141+0000","last_active":"2024-03-12T03:22:34.283358+0000","last_peered":"2024-03-12T03:22:34.283358+0000","last_clean":"2024-03-12T03:22:34.283358+0000","last_became_active":"2024-03-12T03:22:08.210739+0000","last_became_peered":"2024-03-12T03:22:08.210739+0000","last_unstale":"2024-03-12T03:22:34.283358+0000","last_undegraded":"2024-03-12T03:22:34.283358+0000","last_fullsized":"2024-03-12T03:22:34.283358+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:50:22.461527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284123+0000","last_change":"2024-03-12T03:22:04.196964+0000","last_active":"2024-03-12T03:22:34.284123+0000","last_peered":"2024-03-12T03:22:34.284123+0000","last_clean":"2024-03-12T03:22:34.284123+0000","last_became_active":"2024-03-12T03:22:04.196566+0000","last_became_peered":"2024-03-12T03:22:04.196566+0000","last_unstale":"2024-03-12T03:22:34.284123+0000","last_undegraded":"2024-03-12T03:22:34.284123+0000","last_fullsized":"2024-03-12T03:22:34.284123+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:10:24.434895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'18","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285884+0000","last_change":"2024-03-12T03:22:06.198486+0000","last_active":"2024-03-12T03:22:34.285884+0000","last_peered":"2024-03-12T03:22:34.285884+0000","last_clean":"2024-03-12T03:22:34.285884+0000","last_became_active":"2024-03-12T03:22:06.198204+0000","last_became_peered":"2024-03-12T03:22:06.198204+0000","last_unstale":"2024-03-12T03:22:34.285884+0000","last_undegraded":"2024-03-12T03:22:34.285884+0000","last_fullsized":"2024-03-12T03:22:34.285884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:06.709158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284164+0000","last_change":"2024-03-12T03:22:10.223020+0000","last_active":"2024-03-12T03:22:34.284164+0000","last_peered":"2024-03-12T03:22:34.284164+0000","last_clean":"2024-03-12T03:22:34.284164+0000","last_became_active":"2024-03-12T03:22:10.222283+0000","last_became_peered":"2024-03-12T03:22:10.222283+0000","last_unstale":"2024-03-12T03:22:34.284164+0000","last_undegraded":"2024-03-12T03:22:34.284164+0000","last_fullsized":"2024-03-12T03:22:34.284164+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:02:08.852377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284790+0000","last_change":"2024-03-12T03:22:08.206957+0000","last_active":"2024-03-12T03:22:34.284790+0000","last_peered":"2024-03-12T03:22:34.284790+0000","last_clean":"2024-03-12T03:22:34.284790+0000","last_became_active":"2024-03-12T03:22:08.206656+0000","last_became_peered":"2024-03-12T03:22:08.206656+0000","last_unstale":"2024-03-12T03:22:34.284790+0000","last_undegraded":"2024-03-12T03:22:34.284790+0000","last_fullsized":"2024-03-12T03:22:34.284790+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:29:19.283427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284594+0000","last_change":"2024-03-12T03:22:04.196461+0000","last_active":"2024-03-12T03:22:34.284594+0000","last_peered":"2024-03-12T03:22:34.284594+0000","last_clean":"2024-03-12T03:22:34.284594+0000","last_became_active":"2024-03-12T03:22:04.196054+0000","last_became_peered":"2024-03-12T03:22:04.196054+0000","last_unstale":"2024-03-12T03:22:34.284594+0000","last_undegraded":"2024-03-12T03:22:34.284594+0000","last_fullsized":"2024-03-12T03:22:34.284594+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:29:42.946799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285242+0000","last_change":"2024-03-12T03:22:06.198273+0000","last_active":"2024-03-12T03:22:34.285242+0000","last_peered":"2024-03-12T03:22:34.285242+0000","last_clean":"2024-03-12T03:22:34.285242+0000","last_became_active":"2024-03-12T03:22:06.197967+0000","last_became_peered":"2024-03-12T03:22:06.197967+0000","last_unstale":"2024-03-12T03:22:34.285242+0000","last_undegraded":"2024-03-12T03:22:34.285242+0000","last_fullsized":"2024-03-12T03:22:34.285242+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:11.331615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283826+0000","last_change":"2024-03-12T03:22:10.227158+0000","last_active":"2024-03-12T03:22:34.283826+0000","last_peered":"2024-03-12T03:22:34.283826+0000","last_clean":"2024-03-12T03:22:34.283826+0000","last_became_active":"2024-03-12T03:22:10.226765+0000","last_became_peered":"2024-03-12T03:22:10.226765+0000","last_unstale":"2024-03-12T03:22:34.283826+0000","last_undegraded":"2024-03-12T03:22:34.283826+0000","last_fullsized":"2024-03-12T03:22:34.283826+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:16:31.982333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284050+0000","last_change":"2024-03-12T03:22:08.207411+0000","last_active":"2024-03-12T03:22:34.284050+0000","last_peered":"2024-03-12T03:22:34.284050+0000","last_clean":"2024-03-12T03:22:34.284050+0000","last_became_active":"2024-03-12T03:22:08.206746+0000","last_became_peered":"2024-03-12T03:22:08.206746+0000","last_unstale":"2024-03-12T03:22:34.284050+0000","last_undegraded":"2024-03-12T03:22:34.284050+0000","last_fullsized":"2024-03-12T03:22:34.284050+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:34:13.137776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283787+0000","last_change":"2024-03-12T03:22:04.196906+0000","last_active":"2024-03-12T03:22:34.283787+0000","last_peered":"2024-03-12T03:22:34.283787+0000","last_clean":"2024-03-12T03:22:34.283787+0000","last_became_active":"2024-03-12T03:22:04.196234+0000","last_became_peered":"2024-03-12T03:22:04.196234+0000","last_unstale":"2024-03-12T03:22:34.283787+0000","last_undegraded":"2024-03-12T03:22:34.283787+0000","last_fullsized":"2024-03-12T03:22:34.283787+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:36.200162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284331+0000","last_change":"2024-03-12T03:22:06.198460+0000","last_active":"2024-03-12T03:22:34.284331+0000","last_peered":"2024-03-12T03:22:34.284331+0000","last_clean":"2024-03-12T03:22:34.284331+0000","last_became_active":"2024-03-12T03:22:06.198120+0000","last_became_peered":"2024-03-12T03:22:06.198120+0000","last_unstale":"2024-03-12T03:22:34.284331+0000","last_undegraded":"2024-03-12T03:22:34.284331+0000","last_fullsized":"2024-03-12T03:22:34.284331+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:54:09.671716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282737+0000","last_change":"2024-03-12T03:22:10.224865+0000","last_active":"2024-03-12T03:22:34.282737+0000","last_peered":"2024-03-12T03:22:34.282737+0000","last_clean":"2024-03-12T03:22:34.282737+0000","last_became_active":"2024-03-12T03:22:10.224373+0000","last_became_peered":"2024-03-12T03:22:10.224373+0000","last_unstale":"2024-03-12T03:22:34.282737+0000","last_undegraded":"2024-03-12T03:22:34.282737+0000","last_fullsized":"2024-03-12T03:22:34.282737+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:40:12.750053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:49.251810+0000","last_change":"2024-03-12T03:22:08.207768+0000","last_active":"2024-03-12T03:22:49.251810+0000","last_peered":"2024-03-12T03:22:49.251810+0000","last_clean":"2024-03-12T03:22:49.251810+0000","last_became_active":"2024-03-12T03:22:08.207341+0000","last_became_peered":"2024-03-12T03:22:08.207341+0000","last_unstale":"2024-03-12T03:22:49.251810+0000","last_undegraded":"2024-03-12T03:22:49.251810+0000","last_fullsized":"2024-03-12T03:22:49.251810+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:21:31.849899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'2","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282676+0000","last_change":"2024-03-12T03:22:04.194776+0000","last_active":"2024-03-12T03:22:34.282676+0000","last_peered":"2024-03-12T03:22:34.282676+0000","last_clean":"2024-03-12T03:22:34.282676+0000","last_became_active":"2024-03-12T03:22:04.193915+0000","last_became_peered":"2024-03-12T03:22:04.193915+0000","last_unstale":"2024-03-12T03:22:34.282676+0000","last_undegraded":"2024-03-12T03:22:34.282676+0000","last_fullsized":"2024-03-12T03:22:34.282676+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:38:49.259438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.281650+0000","last_change":"2024-03-12T03:22:06.200046+0000","last_active":"2024-03-12T03:22:34.281650+0000","last_peered":"2024-03-12T03:22:34.281650+0000","last_clean":"2024-03-12T03:22:34.281650+0000","last_became_active":"2024-03-12T03:22:06.199644+0000","last_became_peered":"2024-03-12T03:22:06.199644+0000","last_unstale":"2024-03-12T03:22:34.281650+0000","last_undegraded":"2024-03-12T03:22:34.281650+0000","last_fullsized":"2024-03-12T03:22:34.281650+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:45:50.496897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283815+0000","last_change":"2024-03-12T03:22:10.221073+0000","last_active":"2024-03-12T03:22:34.283815+0000","last_peered":"2024-03-12T03:22:34.283815+0000","last_clean":"2024-03-12T03:22:34.283815+0000","last_became_active":"2024-03-12T03:22:10.220435+0000","last_became_peered":"2024-03-12T03:22:10.220435+0000","last_unstale":"2024-03-12T03:22:34.283815+0000","last_undegraded":"2024-03-12T03:22:34.283815+0000","last_fullsized":"2024-03-12T03:22:34.283815+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:59:32.632603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283617+0000","last_change":"2024-03-12T03:22:08.209620+0000","last_active":"2024-03-12T03:22:34.283617+0000","last_peered":"2024-03-12T03:22:34.283617+0000","last_clean":"2024-03-12T03:22:34.283617+0000","last_became_active":"2024-03-12T03:22:08.209237+0000","last_became_peered":"2024-03-12T03:22:08.209237+0000","last_unstale":"2024-03-12T03:22:34.283617+0000","last_undegraded":"2024-03-12T03:22:34.283617+0000","last_fullsized":"2024-03-12T03:22:34.283617+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:37:22.498047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282841+0000","last_change":"2024-03-12T03:22:04.196641+0000","last_active":"2024-03-12T03:22:34.282841+0000","last_peered":"2024-03-12T03:22:34.282841+0000","last_clean":"2024-03-12T03:22:34.282841+0000","last_became_active":"2024-03-12T03:22:04.195928+0000","last_became_peered":"2024-03-12T03:22:04.195928+0000","last_unstale":"2024-03-12T03:22:34.282841+0000","last_undegraded":"2024-03-12T03:22:34.282841+0000","last_fullsized":"2024-03-12T03:22:34.282841+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:28:00.218039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'39","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282272+0000","last_change":"2024-03-12T03:22:06.197752+0000","last_active":"2024-03-12T03:22:34.282272+0000","last_peered":"2024-03-12T03:22:34.282272+0000","last_clean":"2024-03-12T03:22:34.282272+0000","last_became_active":"2024-03-12T03:22:06.197456+0000","last_became_peered":"2024-03-12T03:22:06.197456+0000","last_unstale":"2024-03-12T03:22:34.282272+0000","last_undegraded":"2024-03-12T03:22:34.282272+0000","last_fullsized":"2024-03-12T03:22:34.282272+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:44:43.612133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283966+0000","last_change":"2024-03-12T03:22:10.225145+0000","last_active":"2024-03-12T03:22:34.283966+0000","last_peered":"2024-03-12T03:22:34.283966+0000","last_clean":"2024-03-12T03:22:34.283966+0000","last_became_active":"2024-03-12T03:22:10.223608+0000","last_became_peered":"2024-03-12T03:22:10.223608+0000","last_unstale":"2024-03-12T03:22:34.283966+0000","last_undegraded":"2024-03-12T03:22:34.283966+0000","last_fullsized":"2024-03-12T03:22:34.283966+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:45:08.817362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282295+0000","last_change":"2024-03-12T03:22:08.208788+0000","last_active":"2024-03-12T03:22:34.282295+0000","last_peered":"2024-03-12T03:22:34.282295+0000","last_clean":"2024-03-12T03:22:34.282295+0000","last_became_active":"2024-03-12T03:22:08.208494+0000","last_became_peered":"2024-03-12T03:22:08.208494+0000","last_unstale":"2024-03-12T03:22:34.282295+0000","last_undegraded":"2024-03-12T03:22:34.282295+0000","last_fullsized":"2024-03-12T03:22:34.282295+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:40:32.302062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285801+0000","last_change":"2024-03-12T03:22:04.197334+0000","last_active":"2024-03-12T03:22:34.285801+0000","last_peered":"2024-03-12T03:22:34.285801+0000","last_clean":"2024-03-12T03:22:34.285801+0000","last_became_active":"2024-03-12T03:22:04.196967+0000","last_became_peered":"2024-03-12T03:22:04.196967+0000","last_unstale":"2024-03-12T03:22:34.285801+0000","last_undegraded":"2024-03-12T03:22:34.285801+0000","last_fullsized":"2024-03-12T03:22:34.285801+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:43:22.641580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285859+0000","last_change":"2024-03-12T03:22:06.201766+0000","last_active":"2024-03-12T03:22:34.285859+0000","last_peered":"2024-03-12T03:22:34.285859+0000","last_clean":"2024-03-12T03:22:34.285859+0000","last_became_active":"2024-03-12T03:22:06.201118+0000","last_became_peered":"2024-03-12T03:22:06.201118+0000","last_unstale":"2024-03-12T03:22:34.285859+0000","last_undegraded":"2024-03-12T03:22:34.285859+0000","last_fullsized":"2024-03-12T03:22:34.285859+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:21:59.634609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284119+0000","last_change":"2024-03-12T03:22:10.223423+0000","last_active":"2024-03-12T03:22:34.284119+0000","last_peered":"2024-03-12T03:22:34.284119+0000","last_clean":"2024-03-12T03:22:34.284119+0000","last_became_active":"2024-03-12T03:22:10.222633+0000","last_became_peered":"2024-03-12T03:22:10.222633+0000","last_unstale":"2024-03-12T03:22:34.284119+0000","last_undegraded":"2024-03-12T03:22:34.284119+0000","last_fullsized":"2024-03-12T03:22:34.284119+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:57.634606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282027+0000","last_change":"2024-03-12T03:22:08.208439+0000","last_active":"2024-03-12T03:22:34.282027+0000","last_peered":"2024-03-12T03:22:34.282027+0000","last_clean":"2024-03-12T03:22:34.282027+0000","last_became_active":"2024-03-12T03:22:08.207884+0000","last_became_peered":"2024-03-12T03:22:08.207884+0000","last_unstale":"2024-03-12T03:22:34.282027+0000","last_undegraded":"2024-03-12T03:22:34.282027+0000","last_fullsized":"2024-03-12T03:22:34.282027+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:05:49.542809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284801+0000","last_change":"2024-03-12T03:22:04.199229+0000","last_active":"2024-03-12T03:22:34.284801+0000","last_peered":"2024-03-12T03:22:34.284801+0000","last_clean":"2024-03-12T03:22:34.284801+0000","last_became_active":"2024-03-12T03:22:04.198698+0000","last_became_peered":"2024-03-12T03:22:04.198698+0000","last_unstale":"2024-03-12T03:22:34.284801+0000","last_undegraded":"2024-03-12T03:22:34.284801+0000","last_fullsized":"2024-03-12T03:22:34.284801+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:53.083450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'12","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285010+0000","last_change":"2024-03-12T03:22:06.200651+0000","last_active":"2024-03-12T03:22:34.285010+0000","last_peered":"2024-03-12T03:22:34.285010+0000","last_clean":"2024-03-12T03:22:34.285010+0000","last_became_active":"2024-03-12T03:22:06.200191+0000","last_became_peered":"2024-03-12T03:22:06.200191+0000","last_unstale":"2024-03-12T03:22:34.285010+0000","last_undegraded":"2024-03-12T03:22:34.285010+0000","last_fullsized":"2024-03-12T03:22:34.285010+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:31:26.671903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283051+0000","last_change":"2024-03-12T03:22:10.219336+0000","last_active":"2024-03-12T03:22:34.283051+0000","last_peered":"2024-03-12T03:22:34.283051+0000","last_clean":"2024-03-12T03:22:34.283051+0000","last_became_active":"2024-03-12T03:22:10.219044+0000","last_became_peered":"2024-03-12T03:22:10.219044+0000","last_unstale":"2024-03-12T03:22:34.283051+0000","last_undegraded":"2024-03-12T03:22:34.283051+0000","last_fullsized":"2024-03-12T03:22:34.283051+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:28:43.163283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283922+0000","last_change":"2024-03-12T03:22:08.213789+0000","last_active":"2024-03-12T03:22:34.283922+0000","last_peered":"2024-03-12T03:22:34.283922+0000","last_clean":"2024-03-12T03:22:34.283922+0000","last_became_active":"2024-03-12T03:22:08.213427+0000","last_became_peered":"2024-03-12T03:22:08.213427+0000","last_unstale":"2024-03-12T03:22:34.283922+0000","last_undegraded":"2024-03-12T03:22:34.283922+0000","last_fullsized":"2024-03-12T03:22:34.283922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:04:10.822142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285847+0000","last_change":"2024-03-12T03:22:04.201431+0000","last_active":"2024-03-12T03:22:34.285847+0000","last_peered":"2024-03-12T03:22:34.285847+0000","last_clean":"2024-03-12T03:22:34.285847+0000","last_became_active":"2024-03-12T03:22:04.200577+0000","last_became_peered":"2024-03-12T03:22:04.200577+0000","last_unstale":"2024-03-12T03:22:34.285847+0000","last_undegraded":"2024-03-12T03:22:34.285847+0000","last_fullsized":"2024-03-12T03:22:34.285847+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:08:40.505644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'22","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285814+0000","last_change":"2024-03-12T03:22:06.202040+0000","last_active":"2024-03-12T03:22:34.285814+0000","last_peered":"2024-03-12T03:22:34.285814+0000","last_clean":"2024-03-12T03:22:34.285814+0000","last_became_active":"2024-03-12T03:22:06.200661+0000","last_became_peered":"2024-03-12T03:22:06.200661+0000","last_unstale":"2024-03-12T03:22:34.285814+0000","last_undegraded":"2024-03-12T03:22:34.285814+0000","last_fullsized":"2024-03-12T03:22:34.285814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:56:00.996119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283046+0000","last_change":"2024-03-12T03:22:10.219519+0000","last_active":"2024-03-12T03:22:34.283046+0000","last_peered":"2024-03-12T03:22:34.283046+0000","last_clean":"2024-03-12T03:22:34.283046+0000","last_became_active":"2024-03-12T03:22:10.219225+0000","last_became_peered":"2024-03-12T03:22:10.219225+0000","last_unstale":"2024-03-12T03:22:34.283046+0000","last_undegraded":"2024-03-12T03:22:34.283046+0000","last_fullsized":"2024-03-12T03:22:34.283046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:32:12.759836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.281955+0000","last_change":"2024-03-12T03:22:08.209479+0000","last_active":"2024-03-12T03:22:34.281955+0000","last_peered":"2024-03-12T03:22:34.281955+0000","last_clean":"2024-03-12T03:22:34.281955+0000","last_became_active":"2024-03-12T03:22:08.209132+0000","last_became_peered":"2024-03-12T03:22:08.209132+0000","last_unstale":"2024-03-12T03:22:34.281955+0000","last_undegraded":"2024-03-12T03:22:34.281955+0000","last_fullsized":"2024-03-12T03:22:34.281955+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:15:14.339525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284953+0000","last_change":"2024-03-12T03:22:04.196761+0000","last_active":"2024-03-12T03:22:34.284953+0000","last_peered":"2024-03-12T03:22:34.284953+0000","last_clean":"2024-03-12T03:22:34.284953+0000","last_became_active":"2024-03-12T03:22:04.195988+0000","last_became_peered":"2024-03-12T03:22:04.195988+0000","last_unstale":"2024-03-12T03:22:34.284953+0000","last_undegraded":"2024-03-12T03:22:34.284953+0000","last_fullsized":"2024-03-12T03:22:34.284953+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:06:12.033709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285156+0000","last_change":"2024-03-12T03:22:06.198302+0000","last_active":"2024-03-12T03:22:34.285156+0000","last_peered":"2024-03-12T03:22:34.285156+0000","last_clean":"2024-03-12T03:22:34.285156+0000","last_became_active":"2024-03-12T03:22:06.198019+0000","last_became_peered":"2024-03-12T03:22:06.198019+0000","last_unstale":"2024-03-12T03:22:34.285156+0000","last_undegraded":"2024-03-12T03:22:34.285156+0000","last_fullsized":"2024-03-12T03:22:34.285156+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:36:25.908442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282735+0000","last_change":"2024-03-12T03:22:10.226743+0000","last_active":"2024-03-12T03:22:34.282735+0000","last_peered":"2024-03-12T03:22:34.282735+0000","last_clean":"2024-03-12T03:22:34.282735+0000","last_became_active":"2024-03-12T03:22:10.226404+0000","last_became_peered":"2024-03-12T03:22:10.226404+0000","last_unstale":"2024-03-12T03:22:34.282735+0000","last_undegraded":"2024-03-12T03:22:34.282735+0000","last_fullsized":"2024-03-12T03:22:34.282735+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:55:04.067051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:49.251786+0000","last_change":"2024-03-12T03:22:08.206112+0000","last_active":"2024-03-12T03:22:49.251786+0000","last_peered":"2024-03-12T03:22:49.251786+0000","last_clean":"2024-03-12T03:22:49.251786+0000","last_became_active":"2024-03-12T03:22:08.205823+0000","last_became_peered":"2024-03-12T03:22:08.205823+0000","last_unstale":"2024-03-12T03:22:49.251786+0000","last_undegraded":"2024-03-12T03:22:49.251786+0000","last_fullsized":"2024-03-12T03:22:49.251786+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:01:15.183270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284089+0000","last_change":"2024-03-12T03:22:04.197027+0000","last_active":"2024-03-12T03:22:34.284089+0000","last_peered":"2024-03-12T03:22:34.284089+0000","last_clean":"2024-03-12T03:22:34.284089+0000","last_became_active":"2024-03-12T03:22:04.196516+0000","last_became_peered":"2024-03-12T03:22:04.196516+0000","last_unstale":"2024-03-12T03:22:34.284089+0000","last_undegraded":"2024-03-12T03:22:34.284089+0000","last_fullsized":"2024-03-12T03:22:34.284089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:41:34.960908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282084+0000","last_change":"2024-03-12T03:22:06.198092+0000","last_active":"2024-03-12T03:22:34.282084+0000","last_peered":"2024-03-12T03:22:34.282084+0000","last_clean":"2024-03-12T03:22:34.282084+0000","last_became_active":"2024-03-12T03:22:06.197749+0000","last_became_peered":"2024-03-12T03:22:06.197749+0000","last_unstale":"2024-03-12T03:22:34.282084+0000","last_undegraded":"2024-03-12T03:22:34.282084+0000","last_fullsized":"2024-03-12T03:22:34.282084+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:33:19.247374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284094+0000","last_change":"2024-03-12T03:22:10.222837+0000","last_active":"2024-03-12T03:22:34.284094+0000","last_peered":"2024-03-12T03:22:34.284094+0000","last_clean":"2024-03-12T03:22:34.284094+0000","last_became_active":"2024-03-12T03:22:10.221309+0000","last_became_peered":"2024-03-12T03:22:10.221309+0000","last_unstale":"2024-03-12T03:22:34.284094+0000","last_undegraded":"2024-03-12T03:22:34.284094+0000","last_fullsized":"2024-03-12T03:22:34.284094+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:34:49.921163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:49.251819+0000","last_change":"2024-03-12T03:22:08.207681+0000","last_active":"2024-03-12T03:22:49.251819+0000","last_peered":"2024-03-12T03:22:49.251819+0000","last_clean":"2024-03-12T03:22:49.251819+0000","last_became_active":"2024-03-12T03:22:08.207383+0000","last_became_peered":"2024-03-12T03:22:08.207383+0000","last_unstale":"2024-03-12T03:22:49.251819+0000","last_undegraded":"2024-03-12T03:22:49.251819+0000","last_fullsized":"2024-03-12T03:22:49.251819+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:35.383940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283410+0000","last_change":"2024-03-12T03:22:04.196322+0000","last_active":"2024-03-12T03:22:34.283410+0000","last_peered":"2024-03-12T03:22:34.283410+0000","last_clean":"2024-03-12T03:22:34.283410+0000","last_became_active":"2024-03-12T03:22:04.195975+0000","last_became_peered":"2024-03-12T03:22:04.195975+0000","last_unstale":"2024-03-12T03:22:34.283410+0000","last_undegraded":"2024-03-12T03:22:34.283410+0000","last_fullsized":"2024-03-12T03:22:34.283410+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:45:09.832141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284520+0000","last_change":"2024-03-12T03:22:06.202505+0000","last_active":"2024-03-12T03:22:34.284520+0000","last_peered":"2024-03-12T03:22:34.284520+0000","last_clean":"2024-03-12T03:22:34.284520+0000","last_became_active":"2024-03-12T03:22:06.202214+0000","last_became_peered":"2024-03-12T03:22:06.202214+0000","last_unstale":"2024-03-12T03:22:34.284520+0000","last_undegraded":"2024-03-12T03:22:34.284520+0000","last_fullsized":"2024-03-12T03:22:34.284520+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:23:10.212460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284329+0000","last_change":"2024-03-12T03:22:10.224846+0000","last_active":"2024-03-12T03:22:34.284329+0000","last_peered":"2024-03-12T03:22:34.284329+0000","last_clean":"2024-03-12T03:22:34.284329+0000","last_became_active":"2024-03-12T03:22:10.223625+0000","last_became_peered":"2024-03-12T03:22:10.223625+0000","last_unstale":"2024-03-12T03:22:34.284329+0000","last_undegraded":"2024-03-12T03:22:34.284329+0000","last_fullsized":"2024-03-12T03:22:34.284329+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:08:39.527494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284778+0000","last_change":"2024-03-12T03:22:08.211240+0000","last_active":"2024-03-12T03:22:34.284778+0000","last_peered":"2024-03-12T03:22:34.284778+0000","last_clean":"2024-03-12T03:22:34.284778+0000","last_became_active":"2024-03-12T03:22:08.210438+0000","last_became_peered":"2024-03-12T03:22:08.210438+0000","last_unstale":"2024-03-12T03:22:34.284778+0000","last_undegraded":"2024-03-12T03:22:34.284778+0000","last_fullsized":"2024-03-12T03:22:34.284778+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:25:32.658435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283481+0000","last_change":"2024-03-12T03:22:04.198349+0000","last_active":"2024-03-12T03:22:34.283481+0000","last_peered":"2024-03-12T03:22:34.283481+0000","last_clean":"2024-03-12T03:22:34.283481+0000","last_became_active":"2024-03-12T03:22:04.197125+0000","last_became_peered":"2024-03-12T03:22:04.197125+0000","last_unstale":"2024-03-12T03:22:34.283481+0000","last_undegraded":"2024-03-12T03:22:34.283481+0000","last_fullsized":"2024-03-12T03:22:34.283481+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:23:52.674949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283033+0000","last_change":"2024-03-12T03:22:06.202983+0000","last_active":"2024-03-12T03:22:34.283033+0000","last_peered":"2024-03-12T03:22:34.283033+0000","last_clean":"2024-03-12T03:22:34.283033+0000","last_became_active":"2024-03-12T03:22:06.202607+0000","last_became_peered":"2024-03-12T03:22:06.202607+0000","last_unstale":"2024-03-12T03:22:34.283033+0000","last_undegraded":"2024-03-12T03:22:34.283033+0000","last_fullsized":"2024-03-12T03:22:34.283033+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:10:13.631185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284487+0000","last_change":"2024-03-12T03:22:10.224609+0000","last_active":"2024-03-12T03:22:34.284487+0000","last_peered":"2024-03-12T03:22:34.284487+0000","last_clean":"2024-03-12T03:22:34.284487+0000","last_became_active":"2024-03-12T03:22:10.224261+0000","last_became_peered":"2024-03-12T03:22:10.224261+0000","last_unstale":"2024-03-12T03:22:34.284487+0000","last_undegraded":"2024-03-12T03:22:34.284487+0000","last_fullsized":"2024-03-12T03:22:34.284487+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:35:00.013097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283707+0000","last_change":"2024-03-12T03:22:08.211172+0000","last_active":"2024-03-12T03:22:34.283707+0000","last_peered":"2024-03-12T03:22:34.283707+0000","last_clean":"2024-03-12T03:22:34.283707+0000","last_became_active":"2024-03-12T03:22:08.210751+0000","last_became_peered":"2024-03-12T03:22:08.210751+0000","last_unstale":"2024-03-12T03:22:34.283707+0000","last_undegraded":"2024-03-12T03:22:34.283707+0000","last_fullsized":"2024-03-12T03:22:34.283707+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:21.498847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282986+0000","last_change":"2024-03-12T03:22:04.197105+0000","last_active":"2024-03-12T03:22:34.282986+0000","last_peered":"2024-03-12T03:22:34.282986+0000","last_clean":"2024-03-12T03:22:34.282986+0000","last_became_active":"2024-03-12T03:22:04.196103+0000","last_became_peered":"2024-03-12T03:22:04.196103+0000","last_unstale":"2024-03-12T03:22:34.282986+0000","last_undegraded":"2024-03-12T03:22:34.282986+0000","last_fullsized":"2024-03-12T03:22:34.282986+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:18:01.851810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283243+0000","last_change":"2024-03-12T03:22:06.198533+0000","last_active":"2024-03-12T03:22:34.283243+0000","last_peered":"2024-03-12T03:22:34.283243+0000","last_clean":"2024-03-12T03:22:34.283243+0000","last_became_active":"2024-03-12T03:22:06.197935+0000","last_became_peered":"2024-03-12T03:22:06.197935+0000","last_unstale":"2024-03-12T03:22:34.283243+0000","last_undegraded":"2024-03-12T03:22:34.283243+0000","last_fullsized":"2024-03-12T03:22:34.283243+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:40:27.237249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283269+0000","last_change":"2024-03-12T03:22:10.224206+0000","last_active":"2024-03-12T03:22:34.283269+0000","last_peered":"2024-03-12T03:22:34.283269+0000","last_clean":"2024-03-12T03:22:34.283269+0000","last_became_active":"2024-03-12T03:22:10.223909+0000","last_became_peered":"2024-03-12T03:22:10.223909+0000","last_unstale":"2024-03-12T03:22:34.283269+0000","last_undegraded":"2024-03-12T03:22:34.283269+0000","last_fullsized":"2024-03-12T03:22:34.283269+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:29.616701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285130+0000","last_change":"2024-03-12T03:22:04.201316+0000","last_active":"2024-03-12T03:22:34.285130+0000","last_peered":"2024-03-12T03:22:34.285130+0000","last_clean":"2024-03-12T03:22:34.285130+0000","last_became_active":"2024-03-12T03:22:04.200809+0000","last_became_peered":"2024-03-12T03:22:04.200809+0000","last_unstale":"2024-03-12T03:22:34.285130+0000","last_undegraded":"2024-03-12T03:22:34.285130+0000","last_fullsized":"2024-03-12T03:22:34.285130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:52:51.411909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285781+0000","last_change":"2024-03-12T03:22:06.198420+0000","last_active":"2024-03-12T03:22:34.285781+0000","last_peered":"2024-03-12T03:22:34.285781+0000","last_clean":"2024-03-12T03:22:34.285781+0000","last_became_active":"2024-03-12T03:22:06.198089+0000","last_became_peered":"2024-03-12T03:22:06.198089+0000","last_unstale":"2024-03-12T03:22:34.285781+0000","last_undegraded":"2024-03-12T03:22:34.285781+0000","last_fullsized":"2024-03-12T03:22:34.285781+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:52:41.762099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:49.251908+0000","last_change":"2024-03-12T03:22:08.206318+0000","last_active":"2024-03-12T03:22:49.251908+0000","last_peered":"2024-03-12T03:22:49.251908+0000","last_clean":"2024-03-12T03:22:49.251908+0000","last_became_active":"2024-03-12T03:22:08.206039+0000","last_became_peered":"2024-03-12T03:22:08.206039+0000","last_unstale":"2024-03-12T03:22:49.251908+0000","last_undegraded":"2024-03-12T03:22:49.251908+0000","last_fullsized":"2024-03-12T03:22:49.251908+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:18:02.241476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.281904+0000","last_change":"2024-03-12T03:22:10.220045+0000","last_active":"2024-03-12T03:22:34.281904+0000","last_peered":"2024-03-12T03:22:34.281904+0000","last_clean":"2024-03-12T03:22:34.281904+0000","last_became_active":"2024-03-12T03:22:10.218364+0000","last_became_peered":"2024-03-12T03:22:10.218364+0000","last_unstale":"2024-03-12T03:22:34.281904+0000","last_undegraded":"2024-03-12T03:22:34.281904+0000","last_fullsized":"2024-03-12T03:22:34.281904+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:34:38.962346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283649+0000","last_change":"2024-03-12T03:22:04.195102+0000","last_active":"2024-03-12T03:22:34.283649+0000","last_peered":"2024-03-12T03:22:34.283649+0000","last_clean":"2024-03-12T03:22:34.283649+0000","last_became_active":"2024-03-12T03:22:04.194497+0000","last_became_peered":"2024-03-12T03:22:04.194497+0000","last_unstale":"2024-03-12T03:22:34.283649+0000","last_undegraded":"2024-03-12T03:22:34.283649+0000","last_fullsized":"2024-03-12T03:22:34.283649+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:34:38.035219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:49.251718+0000","last_change":"2024-03-12T03:22:06.201933+0000","last_active":"2024-03-12T03:22:49.251718+0000","last_peered":"2024-03-12T03:22:49.251718+0000","last_clean":"2024-03-12T03:22:49.251718+0000","last_became_active":"2024-03-12T03:22:06.201325+0000","last_became_peered":"2024-03-12T03:22:06.201325+0000","last_unstale":"2024-03-12T03:22:49.251718+0000","last_undegraded":"2024-03-12T03:22:49.251718+0000","last_fullsized":"2024-03-12T03:22:49.251718+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:46:08.259243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283198+0000","last_change":"2024-03-12T03:22:08.207299+0000","last_active":"2024-03-12T03:22:34.283198+0000","last_peered":"2024-03-12T03:22:34.283198+0000","last_clean":"2024-03-12T03:22:34.283198+0000","last_became_active":"2024-03-12T03:22:08.206458+0000","last_became_peered":"2024-03-12T03:22:08.206458+0000","last_unstale":"2024-03-12T03:22:34.283198+0000","last_undegraded":"2024-03-12T03:22:34.283198+0000","last_fullsized":"2024-03-12T03:22:34.283198+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:24:07.066415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282763+0000","last_change":"2024-03-12T03:22:10.226362+0000","last_active":"2024-03-12T03:22:34.282763+0000","last_peered":"2024-03-12T03:22:34.282763+0000","last_clean":"2024-03-12T03:22:34.282763+0000","last_became_active":"2024-03-12T03:22:10.225984+0000","last_became_peered":"2024-03-12T03:22:10.225984+0000","last_unstale":"2024-03-12T03:22:34.282763+0000","last_undegraded":"2024-03-12T03:22:34.282763+0000","last_fullsized":"2024-03-12T03:22:34.282763+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:34:05.731040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284032+0000","last_change":"2024-03-12T03:22:04.196821+0000","last_active":"2024-03-12T03:22:34.284032+0000","last_peered":"2024-03-12T03:22:34.284032+0000","last_clean":"2024-03-12T03:22:34.284032+0000","last_became_active":"2024-03-12T03:22:04.194671+0000","last_became_peered":"2024-03-12T03:22:04.194671+0000","last_unstale":"2024-03-12T03:22:34.284032+0000","last_undegraded":"2024-03-12T03:22:34.284032+0000","last_fullsized":"2024-03-12T03:22:34.284032+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:37:39.786305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'25","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284718+0000","last_change":"2024-03-12T03:22:06.201043+0000","last_active":"2024-03-12T03:22:34.284718+0000","last_peered":"2024-03-12T03:22:34.284718+0000","last_clean":"2024-03-12T03:22:34.284718+0000","last_became_active":"2024-03-12T03:22:06.200503+0000","last_became_peered":"2024-03-12T03:22:06.200503+0000","last_unstale":"2024-03-12T03:22:34.284718+0000","last_undegraded":"2024-03-12T03:22:34.284718+0000","last_fullsized":"2024-03-12T03:22:34.284718+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:02:43.666211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282778+0000","last_change":"2024-03-12T03:22:08.210798+0000","last_active":"2024-03-12T03:22:34.282778+0000","last_peered":"2024-03-12T03:22:34.282778+0000","last_clean":"2024-03-12T03:22:34.282778+0000","last_became_active":"2024-03-12T03:22:08.210432+0000","last_became_peered":"2024-03-12T03:22:08.210432+0000","last_unstale":"2024-03-12T03:22:34.282778+0000","last_undegraded":"2024-03-12T03:22:34.282778+0000","last_fullsized":"2024-03-12T03:22:34.282778+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:14:18.597046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284368+0000","last_change":"2024-03-12T03:22:10.225026+0000","last_active":"2024-03-12T03:22:34.284368+0000","last_peered":"2024-03-12T03:22:34.284368+0000","last_clean":"2024-03-12T03:22:34.284368+0000","last_became_active":"2024-03-12T03:22:10.224555+0000","last_became_peered":"2024-03-12T03:22:10.224555+0000","last_unstale":"2024-03-12T03:22:34.284368+0000","last_undegraded":"2024-03-12T03:22:34.284368+0000","last_fullsized":"2024-03-12T03:22:34.284368+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:38:46.416155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'30","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282615+0000","last_change":"2024-03-12T03:22:06.199102+0000","last_active":"2024-03-12T03:22:34.282615+0000","last_peered":"2024-03-12T03:22:34.282615+0000","last_clean":"2024-03-12T03:22:34.282615+0000","last_became_active":"2024-03-12T03:22:06.198813+0000","last_became_peered":"2024-03-12T03:22:06.198813+0000","last_unstale":"2024-03-12T03:22:34.282615+0000","last_undegraded":"2024-03-12T03:22:34.282615+0000","last_fullsized":"2024-03-12T03:22:34.282615+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:43:43.674860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282463+0000","last_change":"2024-03-12T03:22:04.195312+0000","last_active":"2024-03-12T03:22:34.282463+0000","last_peered":"2024-03-12T03:22:34.282463+0000","last_clean":"2024-03-12T03:22:34.282463+0000","last_became_active":"2024-03-12T03:22:04.194879+0000","last_became_peered":"2024-03-12T03:22:04.194879+0000","last_unstale":"2024-03-12T03:22:34.282463+0000","last_undegraded":"2024-03-12T03:22:34.282463+0000","last_fullsized":"2024-03-12T03:22:34.282463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:24:35.264530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284963+0000","last_change":"2024-03-12T03:22:08.211589+0000","last_active":"2024-03-12T03:22:34.284963+0000","last_peered":"2024-03-12T03:22:34.284963+0000","last_clean":"2024-03-12T03:22:34.284963+0000","last_became_active":"2024-03-12T03:22:08.210985+0000","last_became_peered":"2024-03-12T03:22:08.210985+0000","last_unstale":"2024-03-12T03:22:34.284963+0000","last_undegraded":"2024-03-12T03:22:34.284963+0000","last_fullsized":"2024-03-12T03:22:34.284963+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:25:00.994271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283556+0000","last_change":"2024-03-12T03:22:10.221345+0000","last_active":"2024-03-12T03:22:34.283556+0000","last_peered":"2024-03-12T03:22:34.283556+0000","last_clean":"2024-03-12T03:22:34.283556+0000","last_became_active":"2024-03-12T03:22:10.221037+0000","last_became_peered":"2024-03-12T03:22:10.221037+0000","last_unstale":"2024-03-12T03:22:34.283556+0000","last_undegraded":"2024-03-12T03:22:34.283556+0000","last_fullsized":"2024-03-12T03:22:34.283556+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:34:37.475961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282079+0000","last_change":"2024-03-12T03:22:06.197744+0000","last_active":"2024-03-12T03:22:34.282079+0000","last_peered":"2024-03-12T03:22:34.282079+0000","last_clean":"2024-03-12T03:22:34.282079+0000","last_became_active":"2024-03-12T03:22:06.197400+0000","last_became_peered":"2024-03-12T03:22:06.197400+0000","last_unstale":"2024-03-12T03:22:34.282079+0000","last_undegraded":"2024-03-12T03:22:34.282079+0000","last_fullsized":"2024-03-12T03:22:34.282079+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:24:40.716729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284226+0000","last_change":"2024-03-12T03:22:04.196599+0000","last_active":"2024-03-12T03:22:34.284226+0000","last_peered":"2024-03-12T03:22:34.284226+0000","last_clean":"2024-03-12T03:22:34.284226+0000","last_became_active":"2024-03-12T03:22:04.195845+0000","last_became_peered":"2024-03-12T03:22:04.195845+0000","last_unstale":"2024-03-12T03:22:34.284226+0000","last_undegraded":"2024-03-12T03:22:34.284226+0000","last_fullsized":"2024-03-12T03:22:34.284226+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:02:00.399884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:36.384612+0000","last_change":"2024-03-12T03:21:49.582258+0000","last_active":"2024-03-12T03:22:36.384612+0000","last_peered":"2024-03-12T03:22:36.384612+0000","last_clean":"2024-03-12T03:22:36.384612+0000","last_became_active":"2024-03-12T03:21:49.581839+0000","last_became_peered":"2024-03-12T03:21:49.581839+0000","last_unstale":"2024-03-12T03:22:36.384612+0000","last_undegraded":"2024-03-12T03:22:36.384612+0000","last_fullsized":"2024-03-12T03:22:36.384612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:21:48.573477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:21:48.573477+0000","last_clean_scrub_stamp":"2024-03-12T03:21:48.573477+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:07:19.213260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284709+0000","last_change":"2024-03-12T03:22:08.211294+0000","last_active":"2024-03-12T03:22:34.284709+0000","last_peered":"2024-03-12T03:22:34.284709+0000","last_clean":"2024-03-12T03:22:34.284709+0000","last_became_active":"2024-03-12T03:22:08.210941+0000","last_became_peered":"2024-03-12T03:22:08.210941+0000","last_unstale":"2024-03-12T03:22:34.284709+0000","last_undegraded":"2024-03-12T03:22:34.284709+0000","last_fullsized":"2024-03-12T03:22:34.284709+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:53:45.896536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283918+0000","last_change":"2024-03-12T03:22:10.220793+0000","last_active":"2024-03-12T03:22:34.283918+0000","last_peered":"2024-03-12T03:22:34.283918+0000","last_clean":"2024-03-12T03:22:34.283918+0000","last_became_active":"2024-03-12T03:22:10.220192+0000","last_became_peered":"2024-03-12T03:22:10.220192+0000","last_unstale":"2024-03-12T03:22:34.283918+0000","last_undegraded":"2024-03-12T03:22:34.283918+0000","last_fullsized":"2024-03-12T03:22:34.283918+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:03:05.188945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'20","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282852+0000","last_change":"2024-03-12T03:22:06.198127+0000","last_active":"2024-03-12T03:22:34.282852+0000","last_peered":"2024-03-12T03:22:34.282852+0000","last_clean":"2024-03-12T03:22:34.282852+0000","last_became_active":"2024-03-12T03:22:06.197617+0000","last_became_peered":"2024-03-12T03:22:06.197617+0000","last_unstale":"2024-03-12T03:22:34.282852+0000","last_undegraded":"2024-03-12T03:22:34.282852+0000","last_fullsized":"2024-03-12T03:22:34.282852+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:18.174161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283164+0000","last_change":"2024-03-12T03:22:04.198749+0000","last_active":"2024-03-12T03:22:34.283164+0000","last_peered":"2024-03-12T03:22:34.283164+0000","last_clean":"2024-03-12T03:22:34.283164+0000","last_became_active":"2024-03-12T03:22:04.197328+0000","last_became_peered":"2024-03-12T03:22:04.197328+0000","last_unstale":"2024-03-12T03:22:34.283164+0000","last_undegraded":"2024-03-12T03:22:34.283164+0000","last_fullsized":"2024-03-12T03:22:34.283164+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:25:06.158061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284719+0000","last_change":"2024-03-12T03:22:08.206070+0000","last_active":"2024-03-12T03:22:34.284719+0000","last_peered":"2024-03-12T03:22:34.284719+0000","last_clean":"2024-03-12T03:22:34.284719+0000","last_became_active":"2024-03-12T03:22:08.205678+0000","last_became_peered":"2024-03-12T03:22:08.205678+0000","last_unstale":"2024-03-12T03:22:34.284719+0000","last_undegraded":"2024-03-12T03:22:34.284719+0000","last_fullsized":"2024-03-12T03:22:34.284719+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:47:40.986036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283683+0000","last_change":"2024-03-12T03:22:10.220474+0000","last_active":"2024-03-12T03:22:34.283683+0000","last_peered":"2024-03-12T03:22:34.283683+0000","last_clean":"2024-03-12T03:22:34.283683+0000","last_became_active":"2024-03-12T03:22:10.219971+0000","last_became_peered":"2024-03-12T03:22:10.219971+0000","last_unstale":"2024-03-12T03:22:34.283683+0000","last_undegraded":"2024-03-12T03:22:34.283683+0000","last_fullsized":"2024-03-12T03:22:34.283683+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:12:32.284434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285798+0000","last_change":"2024-03-12T03:22:04.201436+0000","last_active":"2024-03-12T03:22:34.285798+0000","last_peered":"2024-03-12T03:22:34.285798+0000","last_clean":"2024-03-12T03:22:34.285798+0000","last_became_active":"2024-03-12T03:22:04.200929+0000","last_became_peered":"2024-03-12T03:22:04.200929+0000","last_unstale":"2024-03-12T03:22:34.285798+0000","last_undegraded":"2024-03-12T03:22:34.285798+0000","last_fullsized":"2024-03-12T03:22:34.285798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:29:32.733354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'34","reported_seq":94,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:49.251581+0000","last_change":"2024-03-12T03:22:06.199637+0000","last_active":"2024-03-12T03:22:49.251581+0000","last_peered":"2024-03-12T03:22:49.251581+0000","last_clean":"2024-03-12T03:22:49.251581+0000","last_became_active":"2024-03-12T03:22:06.199323+0000","last_became_peered":"2024-03-12T03:22:06.199323+0000","last_unstale":"2024-03-12T03:22:49.251581+0000","last_undegraded":"2024-03-12T03:22:49.251581+0000","last_fullsized":"2024-03-12T03:22:49.251581+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:41:26.795590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:49.251779+0000","last_change":"2024-03-12T03:22:08.207528+0000","last_active":"2024-03-12T03:22:49.251779+0000","last_peered":"2024-03-12T03:22:49.251779+0000","last_clean":"2024-03-12T03:22:49.251779+0000","last_became_active":"2024-03-12T03:22:08.207232+0000","last_became_peered":"2024-03-12T03:22:08.207232+0000","last_unstale":"2024-03-12T03:22:49.251779+0000","last_undegraded":"2024-03-12T03:22:49.251779+0000","last_fullsized":"2024-03-12T03:22:49.251779+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:53.361024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283035+0000","last_change":"2024-03-12T03:22:10.224665+0000","last_active":"2024-03-12T03:22:34.283035+0000","last_peered":"2024-03-12T03:22:34.283035+0000","last_clean":"2024-03-12T03:22:34.283035+0000","last_became_active":"2024-03-12T03:22:10.223400+0000","last_became_peered":"2024-03-12T03:22:10.223400+0000","last_unstale":"2024-03-12T03:22:34.283035+0000","last_undegraded":"2024-03-12T03:22:34.283035+0000","last_fullsized":"2024-03-12T03:22:34.283035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:55:17.460790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284316+0000","last_change":"2024-03-12T03:22:04.194555+0000","last_active":"2024-03-12T03:22:34.284316+0000","last_peered":"2024-03-12T03:22:34.284316+0000","last_clean":"2024-03-12T03:22:34.284316+0000","last_became_active":"2024-03-12T03:22:04.193871+0000","last_became_peered":"2024-03-12T03:22:04.193871+0000","last_unstale":"2024-03-12T03:22:34.284316+0000","last_undegraded":"2024-03-12T03:22:34.284316+0000","last_fullsized":"2024-03-12T03:22:34.284316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:06:08.226618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'19","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285571+0000","last_change":"2024-03-12T03:22:06.201091+0000","last_active":"2024-03-12T03:22:34.285571+0000","last_peered":"2024-03-12T03:22:34.285571+0000","last_clean":"2024-03-12T03:22:34.285571+0000","last_became_active":"2024-03-12T03:22:06.200716+0000","last_became_peered":"2024-03-12T03:22:06.200716+0000","last_unstale":"2024-03-12T03:22:34.285571+0000","last_undegraded":"2024-03-12T03:22:34.285571+0000","last_fullsized":"2024-03-12T03:22:34.285571+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:14:56.705991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:49.251577+0000","last_change":"2024-03-12T03:22:08.209196+0000","last_active":"2024-03-12T03:22:49.251577+0000","last_peered":"2024-03-12T03:22:49.251577+0000","last_clean":"2024-03-12T03:22:49.251577+0000","last_became_active":"2024-03-12T03:22:08.208819+0000","last_became_peered":"2024-03-12T03:22:08.208819+0000","last_unstale":"2024-03-12T03:22:49.251577+0000","last_undegraded":"2024-03-12T03:22:49.251577+0000","last_fullsized":"2024-03-12T03:22:49.251577+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:19.930661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283262+0000","last_change":"2024-03-12T03:22:10.224180+0000","last_active":"2024-03-12T03:22:34.283262+0000","last_peered":"2024-03-12T03:22:34.283262+0000","last_clean":"2024-03-12T03:22:34.283262+0000","last_became_active":"2024-03-12T03:22:10.223890+0000","last_became_peered":"2024-03-12T03:22:10.223890+0000","last_unstale":"2024-03-12T03:22:34.283262+0000","last_undegraded":"2024-03-12T03:22:34.283262+0000","last_fullsized":"2024-03-12T03:22:34.283262+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:33:28.811259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284751+0000","last_change":"2024-03-12T03:22:04.197435+0000","last_active":"2024-03-12T03:22:34.284751+0000","last_peered":"2024-03-12T03:22:34.284751+0000","last_clean":"2024-03-12T03:22:34.284751+0000","last_became_active":"2024-03-12T03:22:04.196633+0000","last_became_peered":"2024-03-12T03:22:04.196633+0000","last_unstale":"2024-03-12T03:22:34.284751+0000","last_undegraded":"2024-03-12T03:22:34.284751+0000","last_fullsized":"2024-03-12T03:22:34.284751+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:15:16.765568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283860+0000","last_change":"2024-03-12T03:22:06.198330+0000","last_active":"2024-03-12T03:22:34.283860+0000","last_peered":"2024-03-12T03:22:34.283860+0000","last_clean":"2024-03-12T03:22:34.283860+0000","last_became_active":"2024-03-12T03:22:06.197900+0000","last_became_peered":"2024-03-12T03:22:06.197900+0000","last_unstale":"2024-03-12T03:22:34.283860+0000","last_undegraded":"2024-03-12T03:22:34.283860+0000","last_fullsized":"2024-03-12T03:22:34.283860+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:02:20.974220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282992+0000","last_change":"2024-03-12T03:22:08.206148+0000","last_active":"2024-03-12T03:22:34.282992+0000","last_peered":"2024-03-12T03:22:34.282992+0000","last_clean":"2024-03-12T03:22:34.282992+0000","last_became_active":"2024-03-12T03:22:08.205652+0000","last_became_peered":"2024-03-12T03:22:08.205652+0000","last_unstale":"2024-03-12T03:22:34.282992+0000","last_undegraded":"2024-03-12T03:22:34.282992+0000","last_fullsized":"2024-03-12T03:22:34.282992+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:10:44.789210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284152+0000","last_change":"2024-03-12T03:22:10.220570+0000","last_active":"2024-03-12T03:22:34.284152+0000","last_peered":"2024-03-12T03:22:34.284152+0000","last_clean":"2024-03-12T03:22:34.284152+0000","last_became_active":"2024-03-12T03:22:10.219865+0000","last_became_peered":"2024-03-12T03:22:10.219865+0000","last_unstale":"2024-03-12T03:22:34.284152+0000","last_undegraded":"2024-03-12T03:22:34.284152+0000","last_fullsized":"2024-03-12T03:22:34.284152+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:06:01.434470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:49.251637+0000","last_change":"2024-03-12T03:22:08.212329+0000","last_active":"2024-03-12T03:22:49.251637+0000","last_peered":"2024-03-12T03:22:49.251637+0000","last_clean":"2024-03-12T03:22:49.251637+0000","last_became_active":"2024-03-12T03:22:08.211998+0000","last_became_peered":"2024-03-12T03:22:08.211998+0000","last_unstale":"2024-03-12T03:22:49.251637+0000","last_undegraded":"2024-03-12T03:22:49.251637+0000","last_fullsized":"2024-03-12T03:22:49.251637+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:39:58.131467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283267+0000","last_change":"2024-03-12T03:22:04.198742+0000","last_active":"2024-03-12T03:22:34.283267+0000","last_peered":"2024-03-12T03:22:34.283267+0000","last_clean":"2024-03-12T03:22:34.283267+0000","last_became_active":"2024-03-12T03:22:04.197470+0000","last_became_peered":"2024-03-12T03:22:04.197470+0000","last_unstale":"2024-03-12T03:22:34.283267+0000","last_undegraded":"2024-03-12T03:22:34.283267+0000","last_fullsized":"2024-03-12T03:22:34.283267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:19:18.414259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'24","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283572+0000","last_change":"2024-03-12T03:22:06.198364+0000","last_active":"2024-03-12T03:22:34.283572+0000","last_peered":"2024-03-12T03:22:34.283572+0000","last_clean":"2024-03-12T03:22:34.283572+0000","last_became_active":"2024-03-12T03:22:06.198035+0000","last_became_peered":"2024-03-12T03:22:06.198035+0000","last_unstale":"2024-03-12T03:22:34.283572+0000","last_undegraded":"2024-03-12T03:22:34.283572+0000","last_fullsized":"2024-03-12T03:22:34.283572+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:37:25.895897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284881+0000","last_change":"2024-03-12T03:22:10.224148+0000","last_active":"2024-03-12T03:22:34.284881+0000","last_peered":"2024-03-12T03:22:34.284881+0000","last_clean":"2024-03-12T03:22:34.284881+0000","last_became_active":"2024-03-12T03:22:10.223838+0000","last_became_peered":"2024-03-12T03:22:10.223838+0000","last_unstale":"2024-03-12T03:22:34.284881+0000","last_undegraded":"2024-03-12T03:22:34.284881+0000","last_fullsized":"2024-03-12T03:22:34.284881+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:50:49.806487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284871+0000","last_change":"2024-03-12T03:22:08.211587+0000","last_active":"2024-03-12T03:22:34.284871+0000","last_peered":"2024-03-12T03:22:34.284871+0000","last_clean":"2024-03-12T03:22:34.284871+0000","last_became_active":"2024-03-12T03:22:08.210504+0000","last_became_peered":"2024-03-12T03:22:08.210504+0000","last_unstale":"2024-03-12T03:22:34.284871+0000","last_undegraded":"2024-03-12T03:22:34.284871+0000","last_fullsized":"2024-03-12T03:22:34.284871+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:15.683378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282565+0000","last_change":"2024-03-12T03:22:04.195035+0000","last_active":"2024-03-12T03:22:34.282565+0000","last_peered":"2024-03-12T03:22:34.282565+0000","last_clean":"2024-03-12T03:22:34.282565+0000","last_became_active":"2024-03-12T03:22:04.194004+0000","last_became_peered":"2024-03-12T03:22:04.194004+0000","last_unstale":"2024-03-12T03:22:34.282565+0000","last_undegraded":"2024-03-12T03:22:34.282565+0000","last_fullsized":"2024-03-12T03:22:34.282565+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:27:48.149760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'27","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283059+0000","last_change":"2024-03-12T03:22:06.199995+0000","last_active":"2024-03-12T03:22:34.283059+0000","last_peered":"2024-03-12T03:22:34.283059+0000","last_clean":"2024-03-12T03:22:34.283059+0000","last_became_active":"2024-03-12T03:22:06.198766+0000","last_became_peered":"2024-03-12T03:22:06.198766+0000","last_unstale":"2024-03-12T03:22:34.283059+0000","last_undegraded":"2024-03-12T03:22:34.283059+0000","last_fullsized":"2024-03-12T03:22:34.283059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:46:44.985314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284697+0000","last_change":"2024-03-12T03:22:10.219335+0000","last_active":"2024-03-12T03:22:34.284697+0000","last_peered":"2024-03-12T03:22:34.284697+0000","last_clean":"2024-03-12T03:22:34.284697+0000","last_became_active":"2024-03-12T03:22:10.219119+0000","last_became_peered":"2024-03-12T03:22:10.219119+0000","last_unstale":"2024-03-12T03:22:34.284697+0000","last_undegraded":"2024-03-12T03:22:34.284697+0000","last_fullsized":"2024-03-12T03:22:34.284697+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:11:00.361204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285148+0000","last_change":"2024-03-12T03:22:04.196305+0000","last_active":"2024-03-12T03:22:34.285148+0000","last_peered":"2024-03-12T03:22:34.285148+0000","last_clean":"2024-03-12T03:22:34.285148+0000","last_became_active":"2024-03-12T03:22:04.195909+0000","last_became_peered":"2024-03-12T03:22:04.195909+0000","last_unstale":"2024-03-12T03:22:34.285148+0000","last_undegraded":"2024-03-12T03:22:34.285148+0000","last_fullsized":"2024-03-12T03:22:34.285148+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:04:28.257235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282266+0000","last_change":"2024-03-12T03:22:06.198321+0000","last_active":"2024-03-12T03:22:34.282266+0000","last_peered":"2024-03-12T03:22:34.282266+0000","last_clean":"2024-03-12T03:22:34.282266+0000","last_became_active":"2024-03-12T03:22:06.197900+0000","last_became_peered":"2024-03-12T03:22:06.197900+0000","last_unstale":"2024-03-12T03:22:34.282266+0000","last_undegraded":"2024-03-12T03:22:34.282266+0000","last_fullsized":"2024-03-12T03:22:34.282266+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:55:20.488421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284288+0000","last_change":"2024-03-12T03:22:08.211995+0000","last_active":"2024-03-12T03:22:34.284288+0000","last_peered":"2024-03-12T03:22:34.284288+0000","last_clean":"2024-03-12T03:22:34.284288+0000","last_became_active":"2024-03-12T03:22:08.211455+0000","last_became_peered":"2024-03-12T03:22:08.211455+0000","last_unstale":"2024-03-12T03:22:34.284288+0000","last_undegraded":"2024-03-12T03:22:34.284288+0000","last_fullsized":"2024-03-12T03:22:34.284288+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:15:54.087061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282664+0000","last_change":"2024-03-12T03:22:10.226502+0000","last_active":"2024-03-12T03:22:34.282664+0000","last_peered":"2024-03-12T03:22:34.282664+0000","last_clean":"2024-03-12T03:22:34.282664+0000","last_became_active":"2024-03-12T03:22:10.226176+0000","last_became_peered":"2024-03-12T03:22:10.226176+0000","last_unstale":"2024-03-12T03:22:34.282664+0000","last_undegraded":"2024-03-12T03:22:34.282664+0000","last_fullsized":"2024-03-12T03:22:34.282664+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:25:14.324426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:49.251628+0000","last_change":"2024-03-12T03:22:08.211510+0000","last_active":"2024-03-12T03:22:49.251628+0000","last_peered":"2024-03-12T03:22:49.251628+0000","last_clean":"2024-03-12T03:22:49.251628+0000","last_became_active":"2024-03-12T03:22:08.211148+0000","last_became_peered":"2024-03-12T03:22:08.211148+0000","last_unstale":"2024-03-12T03:22:49.251628+0000","last_undegraded":"2024-03-12T03:22:49.251628+0000","last_fullsized":"2024-03-12T03:22:49.251628+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:13:00.298813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284460+0000","last_change":"2024-03-12T03:22:04.196934+0000","last_active":"2024-03-12T03:22:34.284460+0000","last_peered":"2024-03-12T03:22:34.284460+0000","last_clean":"2024-03-12T03:22:34.284460+0000","last_became_active":"2024-03-12T03:22:04.196138+0000","last_became_peered":"2024-03-12T03:22:04.196138+0000","last_unstale":"2024-03-12T03:22:34.284460+0000","last_undegraded":"2024-03-12T03:22:34.284460+0000","last_fullsized":"2024-03-12T03:22:34.284460+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:11:29.886706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'25","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284151+0000","last_change":"2024-03-12T03:22:06.201126+0000","last_active":"2024-03-12T03:22:34.284151+0000","last_peered":"2024-03-12T03:22:34.284151+0000","last_clean":"2024-03-12T03:22:34.284151+0000","last_became_active":"2024-03-12T03:22:06.200623+0000","last_became_peered":"2024-03-12T03:22:06.200623+0000","last_unstale":"2024-03-12T03:22:34.284151+0000","last_undegraded":"2024-03-12T03:22:34.284151+0000","last_fullsized":"2024-03-12T03:22:34.284151+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:43:02.117846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283724+0000","last_change":"2024-03-12T03:22:10.219473+0000","last_active":"2024-03-12T03:22:34.283724+0000","last_peered":"2024-03-12T03:22:34.283724+0000","last_clean":"2024-03-12T03:22:34.283724+0000","last_became_active":"2024-03-12T03:22:10.219232+0000","last_became_peered":"2024-03-12T03:22:10.219232+0000","last_unstale":"2024-03-12T03:22:34.283724+0000","last_undegraded":"2024-03-12T03:22:34.283724+0000","last_fullsized":"2024-03-12T03:22:34.283724+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:06:34.371955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284428+0000","last_change":"2024-03-12T03:22:04.197232+0000","last_active":"2024-03-12T03:22:34.284428+0000","last_peered":"2024-03-12T03:22:34.284428+0000","last_clean":"2024-03-12T03:22:34.284428+0000","last_became_active":"2024-03-12T03:22:04.196967+0000","last_became_peered":"2024-03-12T03:22:04.196967+0000","last_unstale":"2024-03-12T03:22:34.284428+0000","last_undegraded":"2024-03-12T03:22:34.284428+0000","last_fullsized":"2024-03-12T03:22:34.284428+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:17:47.670181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285931+0000","last_change":"2024-03-12T03:22:06.201892+0000","last_active":"2024-03-12T03:22:34.285931+0000","last_peered":"2024-03-12T03:22:34.285931+0000","last_clean":"2024-03-12T03:22:34.285931+0000","last_became_active":"2024-03-12T03:22:06.201464+0000","last_became_peered":"2024-03-12T03:22:06.201464+0000","last_unstale":"2024-03-12T03:22:34.285931+0000","last_undegraded":"2024-03-12T03:22:34.285931+0000","last_fullsized":"2024-03-12T03:22:34.285931+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:47:15.357161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283520+0000","last_change":"2024-03-12T03:22:08.213520+0000","last_active":"2024-03-12T03:22:34.283520+0000","last_peered":"2024-03-12T03:22:34.283520+0000","last_clean":"2024-03-12T03:22:34.283520+0000","last_became_active":"2024-03-12T03:22:08.213170+0000","last_became_peered":"2024-03-12T03:22:08.213170+0000","last_unstale":"2024-03-12T03:22:34.283520+0000","last_undegraded":"2024-03-12T03:22:34.283520+0000","last_fullsized":"2024-03-12T03:22:34.283520+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:18:14.436804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.281686+0000","last_change":"2024-03-12T03:22:10.220245+0000","last_active":"2024-03-12T03:22:34.281686+0000","last_peered":"2024-03-12T03:22:34.281686+0000","last_clean":"2024-03-12T03:22:34.281686+0000","last_became_active":"2024-03-12T03:22:10.218325+0000","last_became_peered":"2024-03-12T03:22:10.218325+0000","last_unstale":"2024-03-12T03:22:34.281686+0000","last_undegraded":"2024-03-12T03:22:34.281686+0000","last_fullsized":"2024-03-12T03:22:34.281686+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:27:45.735368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285062+0000","last_change":"2024-03-12T03:22:04.201634+0000","last_active":"2024-03-12T03:22:34.285062+0000","last_peered":"2024-03-12T03:22:34.285062+0000","last_clean":"2024-03-12T03:22:34.285062+0000","last_became_active":"2024-03-12T03:22:04.200975+0000","last_became_peered":"2024-03-12T03:22:04.200975+0000","last_unstale":"2024-03-12T03:22:34.285062+0000","last_undegraded":"2024-03-12T03:22:34.285062+0000","last_fullsized":"2024-03-12T03:22:34.285062+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:56:15.238174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285697+0000","last_change":"2024-03-12T03:22:06.203022+0000","last_active":"2024-03-12T03:22:34.285697+0000","last_peered":"2024-03-12T03:22:34.285697+0000","last_clean":"2024-03-12T03:22:34.285697+0000","last_became_active":"2024-03-12T03:22:06.202692+0000","last_became_peered":"2024-03-12T03:22:06.202692+0000","last_unstale":"2024-03-12T03:22:34.285697+0000","last_undegraded":"2024-03-12T03:22:34.285697+0000","last_fullsized":"2024-03-12T03:22:34.285697+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:28:41.561273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282098+0000","last_change":"2024-03-12T03:22:08.209584+0000","last_active":"2024-03-12T03:22:34.282098+0000","last_peered":"2024-03-12T03:22:34.282098+0000","last_clean":"2024-03-12T03:22:34.282098+0000","last_became_active":"2024-03-12T03:22:08.209232+0000","last_became_peered":"2024-03-12T03:22:08.209232+0000","last_unstale":"2024-03-12T03:22:34.282098+0000","last_undegraded":"2024-03-12T03:22:34.282098+0000","last_fullsized":"2024-03-12T03:22:34.282098+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:14:44.071919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283165+0000","last_change":"2024-03-12T03:22:10.222550+0000","last_active":"2024-03-12T03:22:34.283165+0000","last_peered":"2024-03-12T03:22:34.283165+0000","last_clean":"2024-03-12T03:22:34.283165+0000","last_became_active":"2024-03-12T03:22:10.221438+0000","last_became_peered":"2024-03-12T03:22:10.221438+0000","last_unstale":"2024-03-12T03:22:34.283165+0000","last_undegraded":"2024-03-12T03:22:34.283165+0000","last_fullsized":"2024-03-12T03:22:34.283165+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:46:20.811352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284148+0000","last_change":"2024-03-12T03:22:04.194652+0000","last_active":"2024-03-12T03:22:34.284148+0000","last_peered":"2024-03-12T03:22:34.284148+0000","last_clean":"2024-03-12T03:22:34.284148+0000","last_became_active":"2024-03-12T03:22:04.193908+0000","last_became_peered":"2024-03-12T03:22:04.193908+0000","last_unstale":"2024-03-12T03:22:34.284148+0000","last_undegraded":"2024-03-12T03:22:34.284148+0000","last_fullsized":"2024-03-12T03:22:34.284148+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:18:08.389201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284854+0000","last_change":"2024-03-12T03:22:06.203115+0000","last_active":"2024-03-12T03:22:34.284854+0000","last_peered":"2024-03-12T03:22:34.284854+0000","last_clean":"2024-03-12T03:22:34.284854+0000","last_became_active":"2024-03-12T03:22:06.202656+0000","last_became_peered":"2024-03-12T03:22:06.202656+0000","last_unstale":"2024-03-12T03:22:34.284854+0000","last_undegraded":"2024-03-12T03:22:34.284854+0000","last_fullsized":"2024-03-12T03:22:34.284854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:26:11.403057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283163+0000","last_change":"2024-03-12T03:22:08.206158+0000","last_active":"2024-03-12T03:22:34.283163+0000","last_peered":"2024-03-12T03:22:34.283163+0000","last_clean":"2024-03-12T03:22:34.283163+0000","last_became_active":"2024-03-12T03:22:08.205889+0000","last_became_peered":"2024-03-12T03:22:08.205889+0000","last_unstale":"2024-03-12T03:22:34.283163+0000","last_undegraded":"2024-03-12T03:22:34.283163+0000","last_fullsized":"2024-03-12T03:22:34.283163+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:35:06.325638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":587,"ondisk_log_size":587,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411344,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93704692,"statfs":{"total":95990841344,"available":95953604608,"internally_reserved":0,"allocated":1052672,"data_stored":700191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738395,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93704692,"statfs":{"total":95990841344,"available":95953604608,"internally_reserved":0,"allocated":1052672,"data_stored":700191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:23:00.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:23:00.116 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:00 smithi007 ceph-mon[25800]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:00.116 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:00 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:00.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:00 smithi170 ceph-mon[27708]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:00.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:00 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:00.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:00 smithi033 ceph-mon[27552]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:00.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:00 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:00.523 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:23:01.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:01 smithi007 ceph-mon[25800]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:23:01.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:01 smithi170 ceph-mon[27708]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:23:01.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:01 smithi033 ceph-mon[27552]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:23:02.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:02 smithi007 ceph-mon[25800]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:02.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:02 smithi170 ceph-mon[27708]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:02.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:02 smithi033 ceph-mon[27552]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:03.994 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:23:03.996 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-12T03:23:04.311 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:04 smithi007 ceph-mon[25800]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:04.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:04 smithi170 ceph-mon[27708]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:04.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:04 smithi033 ceph-mon[27552]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:04.650 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-03-12T03:23:02.298765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595984,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":104932,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":101819,"kb_avail":281118236,"statfs":{"total":287972524032,"available":287865073664,"internally_reserved":0,"allocated":3158016,"data_stored":2100573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":104262992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002623"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283798+0000","last_change":"2024-03-12T03:22:10.224487+0000","last_active":"2024-03-12T03:22:34.283798+0000","last_peered":"2024-03-12T03:22:34.283798+0000","last_clean":"2024-03-12T03:22:34.283798+0000","last_became_active":"2024-03-12T03:22:10.223135+0000","last_became_peered":"2024-03-12T03:22:10.223135+0000","last_unstale":"2024-03-12T03:22:34.283798+0000","last_undegraded":"2024-03-12T03:22:34.283798+0000","last_fullsized":"2024-03-12T03:22:34.283798+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:23:15.142101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282965+0000","last_change":"2024-03-12T03:22:08.208026+0000","last_active":"2024-03-12T03:22:34.282965+0000","last_peered":"2024-03-12T03:22:34.282965+0000","last_clean":"2024-03-12T03:22:34.282965+0000","last_became_active":"2024-03-12T03:22:08.207641+0000","last_became_peered":"2024-03-12T03:22:08.207641+0000","last_unstale":"2024-03-12T03:22:34.282965+0000","last_undegraded":"2024-03-12T03:22:34.282965+0000","last_fullsized":"2024-03-12T03:22:34.282965+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:53:20.180584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285437+0000","last_change":"2024-03-12T03:22:04.194608+0000","last_active":"2024-03-12T03:22:34.285437+0000","last_peered":"2024-03-12T03:22:34.285437+0000","last_clean":"2024-03-12T03:22:34.285437+0000","last_became_active":"2024-03-12T03:22:04.193837+0000","last_became_peered":"2024-03-12T03:22:04.193837+0000","last_unstale":"2024-03-12T03:22:34.285437+0000","last_undegraded":"2024-03-12T03:22:34.285437+0000","last_fullsized":"2024-03-12T03:22:34.285437+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:00:11.180424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284008+0000","last_change":"2024-03-12T03:22:06.200644+0000","last_active":"2024-03-12T03:22:34.284008+0000","last_peered":"2024-03-12T03:22:34.284008+0000","last_clean":"2024-03-12T03:22:34.284008+0000","last_became_active":"2024-03-12T03:22:06.199826+0000","last_became_peered":"2024-03-12T03:22:06.199826+0000","last_unstale":"2024-03-12T03:22:34.284008+0000","last_undegraded":"2024-03-12T03:22:34.284008+0000","last_fullsized":"2024-03-12T03:22:34.284008+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:09:11.736828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282489+0000","last_change":"2024-03-12T03:22:10.227009+0000","last_active":"2024-03-12T03:22:34.282489+0000","last_peered":"2024-03-12T03:22:34.282489+0000","last_clean":"2024-03-12T03:22:34.282489+0000","last_became_active":"2024-03-12T03:22:10.226585+0000","last_became_peered":"2024-03-12T03:22:10.226585+0000","last_unstale":"2024-03-12T03:22:34.282489+0000","last_undegraded":"2024-03-12T03:22:34.282489+0000","last_fullsized":"2024-03-12T03:22:34.282489+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:05:01.259966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283510+0000","last_change":"2024-03-12T03:22:08.213121+0000","last_active":"2024-03-12T03:22:34.283510+0000","last_peered":"2024-03-12T03:22:34.283510+0000","last_clean":"2024-03-12T03:22:34.283510+0000","last_became_active":"2024-03-12T03:22:08.212605+0000","last_became_peered":"2024-03-12T03:22:08.212605+0000","last_unstale":"2024-03-12T03:22:34.283510+0000","last_undegraded":"2024-03-12T03:22:34.283510+0000","last_fullsized":"2024-03-12T03:22:34.283510+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:16:18.411495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284265+0000","last_change":"2024-03-12T03:22:04.195506+0000","last_active":"2024-03-12T03:22:34.284265+0000","last_peered":"2024-03-12T03:22:34.284265+0000","last_clean":"2024-03-12T03:22:34.284265+0000","last_became_active":"2024-03-12T03:22:04.195068+0000","last_became_peered":"2024-03-12T03:22:04.195068+0000","last_unstale":"2024-03-12T03:22:34.284265+0000","last_undegraded":"2024-03-12T03:22:34.284265+0000","last_fullsized":"2024-03-12T03:22:34.284265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:38:18.952407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284924+0000","last_change":"2024-03-12T03:22:06.200042+0000","last_active":"2024-03-12T03:22:34.284924+0000","last_peered":"2024-03-12T03:22:34.284924+0000","last_clean":"2024-03-12T03:22:34.284924+0000","last_became_active":"2024-03-12T03:22:06.199705+0000","last_became_peered":"2024-03-12T03:22:06.199705+0000","last_unstale":"2024-03-12T03:22:34.284924+0000","last_undegraded":"2024-03-12T03:22:34.284924+0000","last_fullsized":"2024-03-12T03:22:34.284924+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:09:04.186783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282838+0000","last_change":"2024-03-12T03:22:10.218678+0000","last_active":"2024-03-12T03:22:34.282838+0000","last_peered":"2024-03-12T03:22:34.282838+0000","last_clean":"2024-03-12T03:22:34.282838+0000","last_became_active":"2024-03-12T03:22:10.218375+0000","last_became_peered":"2024-03-12T03:22:10.218375+0000","last_unstale":"2024-03-12T03:22:34.282838+0000","last_undegraded":"2024-03-12T03:22:34.282838+0000","last_fullsized":"2024-03-12T03:22:34.282838+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:07:46.506793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283805+0000","last_change":"2024-03-12T03:22:08.212813+0000","last_active":"2024-03-12T03:22:34.283805+0000","last_peered":"2024-03-12T03:22:34.283805+0000","last_clean":"2024-03-12T03:22:34.283805+0000","last_became_active":"2024-03-12T03:22:08.212249+0000","last_became_peered":"2024-03-12T03:22:08.212249+0000","last_unstale":"2024-03-12T03:22:34.283805+0000","last_undegraded":"2024-03-12T03:22:34.283805+0000","last_fullsized":"2024-03-12T03:22:34.283805+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:25:59.775838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285949+0000","last_change":"2024-03-12T03:22:04.201207+0000","last_active":"2024-03-12T03:22:34.285949+0000","last_peered":"2024-03-12T03:22:34.285949+0000","last_clean":"2024-03-12T03:22:34.285949+0000","last_became_active":"2024-03-12T03:22:04.200522+0000","last_became_peered":"2024-03-12T03:22:04.200522+0000","last_unstale":"2024-03-12T03:22:34.285949+0000","last_undegraded":"2024-03-12T03:22:34.285949+0000","last_fullsized":"2024-03-12T03:22:34.285949+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:21.224226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285839+0000","last_change":"2024-03-12T03:22:06.201984+0000","last_active":"2024-03-12T03:22:34.285839+0000","last_peered":"2024-03-12T03:22:34.285839+0000","last_clean":"2024-03-12T03:22:34.285839+0000","last_became_active":"2024-03-12T03:22:06.200775+0000","last_became_peered":"2024-03-12T03:22:06.200775+0000","last_unstale":"2024-03-12T03:22:34.285839+0000","last_undegraded":"2024-03-12T03:22:34.285839+0000","last_fullsized":"2024-03-12T03:22:34.285839+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:25:56.794935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283555+0000","last_change":"2024-03-12T03:22:10.220543+0000","last_active":"2024-03-12T03:22:34.283555+0000","last_peered":"2024-03-12T03:22:34.283555+0000","last_clean":"2024-03-12T03:22:34.283555+0000","last_became_active":"2024-03-12T03:22:10.219924+0000","last_became_peered":"2024-03-12T03:22:10.219924+0000","last_unstale":"2024-03-12T03:22:34.283555+0000","last_undegraded":"2024-03-12T03:22:34.283555+0000","last_fullsized":"2024-03-12T03:22:34.283555+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:49:30.900784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283358+0000","last_change":"2024-03-12T03:22:08.211141+0000","last_active":"2024-03-12T03:22:34.283358+0000","last_peered":"2024-03-12T03:22:34.283358+0000","last_clean":"2024-03-12T03:22:34.283358+0000","last_became_active":"2024-03-12T03:22:08.210739+0000","last_became_peered":"2024-03-12T03:22:08.210739+0000","last_unstale":"2024-03-12T03:22:34.283358+0000","last_undegraded":"2024-03-12T03:22:34.283358+0000","last_fullsized":"2024-03-12T03:22:34.283358+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:50:22.461527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284123+0000","last_change":"2024-03-12T03:22:04.196964+0000","last_active":"2024-03-12T03:22:34.284123+0000","last_peered":"2024-03-12T03:22:34.284123+0000","last_clean":"2024-03-12T03:22:34.284123+0000","last_became_active":"2024-03-12T03:22:04.196566+0000","last_became_peered":"2024-03-12T03:22:04.196566+0000","last_unstale":"2024-03-12T03:22:34.284123+0000","last_undegraded":"2024-03-12T03:22:34.284123+0000","last_fullsized":"2024-03-12T03:22:34.284123+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:10:24.434895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'18","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285884+0000","last_change":"2024-03-12T03:22:06.198486+0000","last_active":"2024-03-12T03:22:34.285884+0000","last_peered":"2024-03-12T03:22:34.285884+0000","last_clean":"2024-03-12T03:22:34.285884+0000","last_became_active":"2024-03-12T03:22:06.198204+0000","last_became_peered":"2024-03-12T03:22:06.198204+0000","last_unstale":"2024-03-12T03:22:34.285884+0000","last_undegraded":"2024-03-12T03:22:34.285884+0000","last_fullsized":"2024-03-12T03:22:34.285884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:06.709158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284164+0000","last_change":"2024-03-12T03:22:10.223020+0000","last_active":"2024-03-12T03:22:34.284164+0000","last_peered":"2024-03-12T03:22:34.284164+0000","last_clean":"2024-03-12T03:22:34.284164+0000","last_became_active":"2024-03-12T03:22:10.222283+0000","last_became_peered":"2024-03-12T03:22:10.222283+0000","last_unstale":"2024-03-12T03:22:34.284164+0000","last_undegraded":"2024-03-12T03:22:34.284164+0000","last_fullsized":"2024-03-12T03:22:34.284164+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:02:08.852377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284790+0000","last_change":"2024-03-12T03:22:08.206957+0000","last_active":"2024-03-12T03:22:34.284790+0000","last_peered":"2024-03-12T03:22:34.284790+0000","last_clean":"2024-03-12T03:22:34.284790+0000","last_became_active":"2024-03-12T03:22:08.206656+0000","last_became_peered":"2024-03-12T03:22:08.206656+0000","last_unstale":"2024-03-12T03:22:34.284790+0000","last_undegraded":"2024-03-12T03:22:34.284790+0000","last_fullsized":"2024-03-12T03:22:34.284790+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:29:19.283427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284594+0000","last_change":"2024-03-12T03:22:04.196461+0000","last_active":"2024-03-12T03:22:34.284594+0000","last_peered":"2024-03-12T03:22:34.284594+0000","last_clean":"2024-03-12T03:22:34.284594+0000","last_became_active":"2024-03-12T03:22:04.196054+0000","last_became_peered":"2024-03-12T03:22:04.196054+0000","last_unstale":"2024-03-12T03:22:34.284594+0000","last_undegraded":"2024-03-12T03:22:34.284594+0000","last_fullsized":"2024-03-12T03:22:34.284594+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:29:42.946799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285242+0000","last_change":"2024-03-12T03:22:06.198273+0000","last_active":"2024-03-12T03:22:34.285242+0000","last_peered":"2024-03-12T03:22:34.285242+0000","last_clean":"2024-03-12T03:22:34.285242+0000","last_became_active":"2024-03-12T03:22:06.197967+0000","last_became_peered":"2024-03-12T03:22:06.197967+0000","last_unstale":"2024-03-12T03:22:34.285242+0000","last_undegraded":"2024-03-12T03:22:34.285242+0000","last_fullsized":"2024-03-12T03:22:34.285242+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:11.331615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283826+0000","last_change":"2024-03-12T03:22:10.227158+0000","last_active":"2024-03-12T03:22:34.283826+0000","last_peered":"2024-03-12T03:22:34.283826+0000","last_clean":"2024-03-12T03:22:34.283826+0000","last_became_active":"2024-03-12T03:22:10.226765+0000","last_became_peered":"2024-03-12T03:22:10.226765+0000","last_unstale":"2024-03-12T03:22:34.283826+0000","last_undegraded":"2024-03-12T03:22:34.283826+0000","last_fullsized":"2024-03-12T03:22:34.283826+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:16:31.982333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284050+0000","last_change":"2024-03-12T03:22:08.207411+0000","last_active":"2024-03-12T03:22:34.284050+0000","last_peered":"2024-03-12T03:22:34.284050+0000","last_clean":"2024-03-12T03:22:34.284050+0000","last_became_active":"2024-03-12T03:22:08.206746+0000","last_became_peered":"2024-03-12T03:22:08.206746+0000","last_unstale":"2024-03-12T03:22:34.284050+0000","last_undegraded":"2024-03-12T03:22:34.284050+0000","last_fullsized":"2024-03-12T03:22:34.284050+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:34:13.137776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283787+0000","last_change":"2024-03-12T03:22:04.196906+0000","last_active":"2024-03-12T03:22:34.283787+0000","last_peered":"2024-03-12T03:22:34.283787+0000","last_clean":"2024-03-12T03:22:34.283787+0000","last_became_active":"2024-03-12T03:22:04.196234+0000","last_became_peered":"2024-03-12T03:22:04.196234+0000","last_unstale":"2024-03-12T03:22:34.283787+0000","last_undegraded":"2024-03-12T03:22:34.283787+0000","last_fullsized":"2024-03-12T03:22:34.283787+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:36.200162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284331+0000","last_change":"2024-03-12T03:22:06.198460+0000","last_active":"2024-03-12T03:22:34.284331+0000","last_peered":"2024-03-12T03:22:34.284331+0000","last_clean":"2024-03-12T03:22:34.284331+0000","last_became_active":"2024-03-12T03:22:06.198120+0000","last_became_peered":"2024-03-12T03:22:06.198120+0000","last_unstale":"2024-03-12T03:22:34.284331+0000","last_undegraded":"2024-03-12T03:22:34.284331+0000","last_fullsized":"2024-03-12T03:22:34.284331+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:54:09.671716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282737+0000","last_change":"2024-03-12T03:22:10.224865+0000","last_active":"2024-03-12T03:22:34.282737+0000","last_peered":"2024-03-12T03:22:34.282737+0000","last_clean":"2024-03-12T03:22:34.282737+0000","last_became_active":"2024-03-12T03:22:10.224373+0000","last_became_peered":"2024-03-12T03:22:10.224373+0000","last_unstale":"2024-03-12T03:22:34.282737+0000","last_undegraded":"2024-03-12T03:22:34.282737+0000","last_fullsized":"2024-03-12T03:22:34.282737+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:40:12.750053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:54.251931+0000","last_change":"2024-03-12T03:22:08.207768+0000","last_active":"2024-03-12T03:22:54.251931+0000","last_peered":"2024-03-12T03:22:54.251931+0000","last_clean":"2024-03-12T03:22:54.251931+0000","last_became_active":"2024-03-12T03:22:08.207341+0000","last_became_peered":"2024-03-12T03:22:08.207341+0000","last_unstale":"2024-03-12T03:22:54.251931+0000","last_undegraded":"2024-03-12T03:22:54.251931+0000","last_fullsized":"2024-03-12T03:22:54.251931+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:21:31.849899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'2","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282676+0000","last_change":"2024-03-12T03:22:04.194776+0000","last_active":"2024-03-12T03:22:34.282676+0000","last_peered":"2024-03-12T03:22:34.282676+0000","last_clean":"2024-03-12T03:22:34.282676+0000","last_became_active":"2024-03-12T03:22:04.193915+0000","last_became_peered":"2024-03-12T03:22:04.193915+0000","last_unstale":"2024-03-12T03:22:34.282676+0000","last_undegraded":"2024-03-12T03:22:34.282676+0000","last_fullsized":"2024-03-12T03:22:34.282676+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:38:49.259438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.281650+0000","last_change":"2024-03-12T03:22:06.200046+0000","last_active":"2024-03-12T03:22:34.281650+0000","last_peered":"2024-03-12T03:22:34.281650+0000","last_clean":"2024-03-12T03:22:34.281650+0000","last_became_active":"2024-03-12T03:22:06.199644+0000","last_became_peered":"2024-03-12T03:22:06.199644+0000","last_unstale":"2024-03-12T03:22:34.281650+0000","last_undegraded":"2024-03-12T03:22:34.281650+0000","last_fullsized":"2024-03-12T03:22:34.281650+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:45:50.496897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283815+0000","last_change":"2024-03-12T03:22:10.221073+0000","last_active":"2024-03-12T03:22:34.283815+0000","last_peered":"2024-03-12T03:22:34.283815+0000","last_clean":"2024-03-12T03:22:34.283815+0000","last_became_active":"2024-03-12T03:22:10.220435+0000","last_became_peered":"2024-03-12T03:22:10.220435+0000","last_unstale":"2024-03-12T03:22:34.283815+0000","last_undegraded":"2024-03-12T03:22:34.283815+0000","last_fullsized":"2024-03-12T03:22:34.283815+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:59:32.632603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283617+0000","last_change":"2024-03-12T03:22:08.209620+0000","last_active":"2024-03-12T03:22:34.283617+0000","last_peered":"2024-03-12T03:22:34.283617+0000","last_clean":"2024-03-12T03:22:34.283617+0000","last_became_active":"2024-03-12T03:22:08.209237+0000","last_became_peered":"2024-03-12T03:22:08.209237+0000","last_unstale":"2024-03-12T03:22:34.283617+0000","last_undegraded":"2024-03-12T03:22:34.283617+0000","last_fullsized":"2024-03-12T03:22:34.283617+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:37:22.498047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282841+0000","last_change":"2024-03-12T03:22:04.196641+0000","last_active":"2024-03-12T03:22:34.282841+0000","last_peered":"2024-03-12T03:22:34.282841+0000","last_clean":"2024-03-12T03:22:34.282841+0000","last_became_active":"2024-03-12T03:22:04.195928+0000","last_became_peered":"2024-03-12T03:22:04.195928+0000","last_unstale":"2024-03-12T03:22:34.282841+0000","last_undegraded":"2024-03-12T03:22:34.282841+0000","last_fullsized":"2024-03-12T03:22:34.282841+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:28:00.218039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'39","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282272+0000","last_change":"2024-03-12T03:22:06.197752+0000","last_active":"2024-03-12T03:22:34.282272+0000","last_peered":"2024-03-12T03:22:34.282272+0000","last_clean":"2024-03-12T03:22:34.282272+0000","last_became_active":"2024-03-12T03:22:06.197456+0000","last_became_peered":"2024-03-12T03:22:06.197456+0000","last_unstale":"2024-03-12T03:22:34.282272+0000","last_undegraded":"2024-03-12T03:22:34.282272+0000","last_fullsized":"2024-03-12T03:22:34.282272+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:44:43.612133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283966+0000","last_change":"2024-03-12T03:22:10.225145+0000","last_active":"2024-03-12T03:22:34.283966+0000","last_peered":"2024-03-12T03:22:34.283966+0000","last_clean":"2024-03-12T03:22:34.283966+0000","last_became_active":"2024-03-12T03:22:10.223608+0000","last_became_peered":"2024-03-12T03:22:10.223608+0000","last_unstale":"2024-03-12T03:22:34.283966+0000","last_undegraded":"2024-03-12T03:22:34.283966+0000","last_fullsized":"2024-03-12T03:22:34.283966+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:45:08.817362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282295+0000","last_change":"2024-03-12T03:22:08.208788+0000","last_active":"2024-03-12T03:22:34.282295+0000","last_peered":"2024-03-12T03:22:34.282295+0000","last_clean":"2024-03-12T03:22:34.282295+0000","last_became_active":"2024-03-12T03:22:08.208494+0000","last_became_peered":"2024-03-12T03:22:08.208494+0000","last_unstale":"2024-03-12T03:22:34.282295+0000","last_undegraded":"2024-03-12T03:22:34.282295+0000","last_fullsized":"2024-03-12T03:22:34.282295+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:40:32.302062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285801+0000","last_change":"2024-03-12T03:22:04.197334+0000","last_active":"2024-03-12T03:22:34.285801+0000","last_peered":"2024-03-12T03:22:34.285801+0000","last_clean":"2024-03-12T03:22:34.285801+0000","last_became_active":"2024-03-12T03:22:04.196967+0000","last_became_peered":"2024-03-12T03:22:04.196967+0000","last_unstale":"2024-03-12T03:22:34.285801+0000","last_undegraded":"2024-03-12T03:22:34.285801+0000","last_fullsized":"2024-03-12T03:22:34.285801+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:43:22.641580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285859+0000","last_change":"2024-03-12T03:22:06.201766+0000","last_active":"2024-03-12T03:22:34.285859+0000","last_peered":"2024-03-12T03:22:34.285859+0000","last_clean":"2024-03-12T03:22:34.285859+0000","last_became_active":"2024-03-12T03:22:06.201118+0000","last_became_peered":"2024-03-12T03:22:06.201118+0000","last_unstale":"2024-03-12T03:22:34.285859+0000","last_undegraded":"2024-03-12T03:22:34.285859+0000","last_fullsized":"2024-03-12T03:22:34.285859+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:21:59.634609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284119+0000","last_change":"2024-03-12T03:22:10.223423+0000","last_active":"2024-03-12T03:22:34.284119+0000","last_peered":"2024-03-12T03:22:34.284119+0000","last_clean":"2024-03-12T03:22:34.284119+0000","last_became_active":"2024-03-12T03:22:10.222633+0000","last_became_peered":"2024-03-12T03:22:10.222633+0000","last_unstale":"2024-03-12T03:22:34.284119+0000","last_undegraded":"2024-03-12T03:22:34.284119+0000","last_fullsized":"2024-03-12T03:22:34.284119+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:57.634606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282027+0000","last_change":"2024-03-12T03:22:08.208439+0000","last_active":"2024-03-12T03:22:34.282027+0000","last_peered":"2024-03-12T03:22:34.282027+0000","last_clean":"2024-03-12T03:22:34.282027+0000","last_became_active":"2024-03-12T03:22:08.207884+0000","last_became_peered":"2024-03-12T03:22:08.207884+0000","last_unstale":"2024-03-12T03:22:34.282027+0000","last_undegraded":"2024-03-12T03:22:34.282027+0000","last_fullsized":"2024-03-12T03:22:34.282027+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:05:49.542809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284801+0000","last_change":"2024-03-12T03:22:04.199229+0000","last_active":"2024-03-12T03:22:34.284801+0000","last_peered":"2024-03-12T03:22:34.284801+0000","last_clean":"2024-03-12T03:22:34.284801+0000","last_became_active":"2024-03-12T03:22:04.198698+0000","last_became_peered":"2024-03-12T03:22:04.198698+0000","last_unstale":"2024-03-12T03:22:34.284801+0000","last_undegraded":"2024-03-12T03:22:34.284801+0000","last_fullsized":"2024-03-12T03:22:34.284801+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:53.083450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'12","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285010+0000","last_change":"2024-03-12T03:22:06.200651+0000","last_active":"2024-03-12T03:22:34.285010+0000","last_peered":"2024-03-12T03:22:34.285010+0000","last_clean":"2024-03-12T03:22:34.285010+0000","last_became_active":"2024-03-12T03:22:06.200191+0000","last_became_peered":"2024-03-12T03:22:06.200191+0000","last_unstale":"2024-03-12T03:22:34.285010+0000","last_undegraded":"2024-03-12T03:22:34.285010+0000","last_fullsized":"2024-03-12T03:22:34.285010+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:31:26.671903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283051+0000","last_change":"2024-03-12T03:22:10.219336+0000","last_active":"2024-03-12T03:22:34.283051+0000","last_peered":"2024-03-12T03:22:34.283051+0000","last_clean":"2024-03-12T03:22:34.283051+0000","last_became_active":"2024-03-12T03:22:10.219044+0000","last_became_peered":"2024-03-12T03:22:10.219044+0000","last_unstale":"2024-03-12T03:22:34.283051+0000","last_undegraded":"2024-03-12T03:22:34.283051+0000","last_fullsized":"2024-03-12T03:22:34.283051+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:28:43.163283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283922+0000","last_change":"2024-03-12T03:22:08.213789+0000","last_active":"2024-03-12T03:22:34.283922+0000","last_peered":"2024-03-12T03:22:34.283922+0000","last_clean":"2024-03-12T03:22:34.283922+0000","last_became_active":"2024-03-12T03:22:08.213427+0000","last_became_peered":"2024-03-12T03:22:08.213427+0000","last_unstale":"2024-03-12T03:22:34.283922+0000","last_undegraded":"2024-03-12T03:22:34.283922+0000","last_fullsized":"2024-03-12T03:22:34.283922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:04:10.822142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285847+0000","last_change":"2024-03-12T03:22:04.201431+0000","last_active":"2024-03-12T03:22:34.285847+0000","last_peered":"2024-03-12T03:22:34.285847+0000","last_clean":"2024-03-12T03:22:34.285847+0000","last_became_active":"2024-03-12T03:22:04.200577+0000","last_became_peered":"2024-03-12T03:22:04.200577+0000","last_unstale":"2024-03-12T03:22:34.285847+0000","last_undegraded":"2024-03-12T03:22:34.285847+0000","last_fullsized":"2024-03-12T03:22:34.285847+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:08:40.505644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'22","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285814+0000","last_change":"2024-03-12T03:22:06.202040+0000","last_active":"2024-03-12T03:22:34.285814+0000","last_peered":"2024-03-12T03:22:34.285814+0000","last_clean":"2024-03-12T03:22:34.285814+0000","last_became_active":"2024-03-12T03:22:06.200661+0000","last_became_peered":"2024-03-12T03:22:06.200661+0000","last_unstale":"2024-03-12T03:22:34.285814+0000","last_undegraded":"2024-03-12T03:22:34.285814+0000","last_fullsized":"2024-03-12T03:22:34.285814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:56:00.996119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283046+0000","last_change":"2024-03-12T03:22:10.219519+0000","last_active":"2024-03-12T03:22:34.283046+0000","last_peered":"2024-03-12T03:22:34.283046+0000","last_clean":"2024-03-12T03:22:34.283046+0000","last_became_active":"2024-03-12T03:22:10.219225+0000","last_became_peered":"2024-03-12T03:22:10.219225+0000","last_unstale":"2024-03-12T03:22:34.283046+0000","last_undegraded":"2024-03-12T03:22:34.283046+0000","last_fullsized":"2024-03-12T03:22:34.283046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:32:12.759836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.281955+0000","last_change":"2024-03-12T03:22:08.209479+0000","last_active":"2024-03-12T03:22:34.281955+0000","last_peered":"2024-03-12T03:22:34.281955+0000","last_clean":"2024-03-12T03:22:34.281955+0000","last_became_active":"2024-03-12T03:22:08.209132+0000","last_became_peered":"2024-03-12T03:22:08.209132+0000","last_unstale":"2024-03-12T03:22:34.281955+0000","last_undegraded":"2024-03-12T03:22:34.281955+0000","last_fullsized":"2024-03-12T03:22:34.281955+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:15:14.339525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284953+0000","last_change":"2024-03-12T03:22:04.196761+0000","last_active":"2024-03-12T03:22:34.284953+0000","last_peered":"2024-03-12T03:22:34.284953+0000","last_clean":"2024-03-12T03:22:34.284953+0000","last_became_active":"2024-03-12T03:22:04.195988+0000","last_became_peered":"2024-03-12T03:22:04.195988+0000","last_unstale":"2024-03-12T03:22:34.284953+0000","last_undegraded":"2024-03-12T03:22:34.284953+0000","last_fullsized":"2024-03-12T03:22:34.284953+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:06:12.033709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'17","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285156+0000","last_change":"2024-03-12T03:22:06.198302+0000","last_active":"2024-03-12T03:22:34.285156+0000","last_peered":"2024-03-12T03:22:34.285156+0000","last_clean":"2024-03-12T03:22:34.285156+0000","last_became_active":"2024-03-12T03:22:06.198019+0000","last_became_peered":"2024-03-12T03:22:06.198019+0000","last_unstale":"2024-03-12T03:22:34.285156+0000","last_undegraded":"2024-03-12T03:22:34.285156+0000","last_fullsized":"2024-03-12T03:22:34.285156+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:36:25.908442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282735+0000","last_change":"2024-03-12T03:22:10.226743+0000","last_active":"2024-03-12T03:22:34.282735+0000","last_peered":"2024-03-12T03:22:34.282735+0000","last_clean":"2024-03-12T03:22:34.282735+0000","last_became_active":"2024-03-12T03:22:10.226404+0000","last_became_peered":"2024-03-12T03:22:10.226404+0000","last_unstale":"2024-03-12T03:22:34.282735+0000","last_undegraded":"2024-03-12T03:22:34.282735+0000","last_fullsized":"2024-03-12T03:22:34.282735+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:55:04.067051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:54.251899+0000","last_change":"2024-03-12T03:22:08.206112+0000","last_active":"2024-03-12T03:22:54.251899+0000","last_peered":"2024-03-12T03:22:54.251899+0000","last_clean":"2024-03-12T03:22:54.251899+0000","last_became_active":"2024-03-12T03:22:08.205823+0000","last_became_peered":"2024-03-12T03:22:08.205823+0000","last_unstale":"2024-03-12T03:22:54.251899+0000","last_undegraded":"2024-03-12T03:22:54.251899+0000","last_fullsized":"2024-03-12T03:22:54.251899+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:01:15.183270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284089+0000","last_change":"2024-03-12T03:22:04.197027+0000","last_active":"2024-03-12T03:22:34.284089+0000","last_peered":"2024-03-12T03:22:34.284089+0000","last_clean":"2024-03-12T03:22:34.284089+0000","last_became_active":"2024-03-12T03:22:04.196516+0000","last_became_peered":"2024-03-12T03:22:04.196516+0000","last_unstale":"2024-03-12T03:22:34.284089+0000","last_undegraded":"2024-03-12T03:22:34.284089+0000","last_fullsized":"2024-03-12T03:22:34.284089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:41:34.960908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282084+0000","last_change":"2024-03-12T03:22:06.198092+0000","last_active":"2024-03-12T03:22:34.282084+0000","last_peered":"2024-03-12T03:22:34.282084+0000","last_clean":"2024-03-12T03:22:34.282084+0000","last_became_active":"2024-03-12T03:22:06.197749+0000","last_became_peered":"2024-03-12T03:22:06.197749+0000","last_unstale":"2024-03-12T03:22:34.282084+0000","last_undegraded":"2024-03-12T03:22:34.282084+0000","last_fullsized":"2024-03-12T03:22:34.282084+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:33:19.247374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284094+0000","last_change":"2024-03-12T03:22:10.222837+0000","last_active":"2024-03-12T03:22:34.284094+0000","last_peered":"2024-03-12T03:22:34.284094+0000","last_clean":"2024-03-12T03:22:34.284094+0000","last_became_active":"2024-03-12T03:22:10.221309+0000","last_became_peered":"2024-03-12T03:22:10.221309+0000","last_unstale":"2024-03-12T03:22:34.284094+0000","last_undegraded":"2024-03-12T03:22:34.284094+0000","last_fullsized":"2024-03-12T03:22:34.284094+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:34:49.921163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:54.251959+0000","last_change":"2024-03-12T03:22:08.207681+0000","last_active":"2024-03-12T03:22:54.251959+0000","last_peered":"2024-03-12T03:22:54.251959+0000","last_clean":"2024-03-12T03:22:54.251959+0000","last_became_active":"2024-03-12T03:22:08.207383+0000","last_became_peered":"2024-03-12T03:22:08.207383+0000","last_unstale":"2024-03-12T03:22:54.251959+0000","last_undegraded":"2024-03-12T03:22:54.251959+0000","last_fullsized":"2024-03-12T03:22:54.251959+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:35.383940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283410+0000","last_change":"2024-03-12T03:22:04.196322+0000","last_active":"2024-03-12T03:22:34.283410+0000","last_peered":"2024-03-12T03:22:34.283410+0000","last_clean":"2024-03-12T03:22:34.283410+0000","last_became_active":"2024-03-12T03:22:04.195975+0000","last_became_peered":"2024-03-12T03:22:04.195975+0000","last_unstale":"2024-03-12T03:22:34.283410+0000","last_undegraded":"2024-03-12T03:22:34.283410+0000","last_fullsized":"2024-03-12T03:22:34.283410+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:45:09.832141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284520+0000","last_change":"2024-03-12T03:22:06.202505+0000","last_active":"2024-03-12T03:22:34.284520+0000","last_peered":"2024-03-12T03:22:34.284520+0000","last_clean":"2024-03-12T03:22:34.284520+0000","last_became_active":"2024-03-12T03:22:06.202214+0000","last_became_peered":"2024-03-12T03:22:06.202214+0000","last_unstale":"2024-03-12T03:22:34.284520+0000","last_undegraded":"2024-03-12T03:22:34.284520+0000","last_fullsized":"2024-03-12T03:22:34.284520+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:23:10.212460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284329+0000","last_change":"2024-03-12T03:22:10.224846+0000","last_active":"2024-03-12T03:22:34.284329+0000","last_peered":"2024-03-12T03:22:34.284329+0000","last_clean":"2024-03-12T03:22:34.284329+0000","last_became_active":"2024-03-12T03:22:10.223625+0000","last_became_peered":"2024-03-12T03:22:10.223625+0000","last_unstale":"2024-03-12T03:22:34.284329+0000","last_undegraded":"2024-03-12T03:22:34.284329+0000","last_fullsized":"2024-03-12T03:22:34.284329+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:08:39.527494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284778+0000","last_change":"2024-03-12T03:22:08.211240+0000","last_active":"2024-03-12T03:22:34.284778+0000","last_peered":"2024-03-12T03:22:34.284778+0000","last_clean":"2024-03-12T03:22:34.284778+0000","last_became_active":"2024-03-12T03:22:08.210438+0000","last_became_peered":"2024-03-12T03:22:08.210438+0000","last_unstale":"2024-03-12T03:22:34.284778+0000","last_undegraded":"2024-03-12T03:22:34.284778+0000","last_fullsized":"2024-03-12T03:22:34.284778+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:25:32.658435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283481+0000","last_change":"2024-03-12T03:22:04.198349+0000","last_active":"2024-03-12T03:22:34.283481+0000","last_peered":"2024-03-12T03:22:34.283481+0000","last_clean":"2024-03-12T03:22:34.283481+0000","last_became_active":"2024-03-12T03:22:04.197125+0000","last_became_peered":"2024-03-12T03:22:04.197125+0000","last_unstale":"2024-03-12T03:22:34.283481+0000","last_undegraded":"2024-03-12T03:22:34.283481+0000","last_fullsized":"2024-03-12T03:22:34.283481+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:23:52.674949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283033+0000","last_change":"2024-03-12T03:22:06.202983+0000","last_active":"2024-03-12T03:22:34.283033+0000","last_peered":"2024-03-12T03:22:34.283033+0000","last_clean":"2024-03-12T03:22:34.283033+0000","last_became_active":"2024-03-12T03:22:06.202607+0000","last_became_peered":"2024-03-12T03:22:06.202607+0000","last_unstale":"2024-03-12T03:22:34.283033+0000","last_undegraded":"2024-03-12T03:22:34.283033+0000","last_fullsized":"2024-03-12T03:22:34.283033+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:10:13.631185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284487+0000","last_change":"2024-03-12T03:22:10.224609+0000","last_active":"2024-03-12T03:22:34.284487+0000","last_peered":"2024-03-12T03:22:34.284487+0000","last_clean":"2024-03-12T03:22:34.284487+0000","last_became_active":"2024-03-12T03:22:10.224261+0000","last_became_peered":"2024-03-12T03:22:10.224261+0000","last_unstale":"2024-03-12T03:22:34.284487+0000","last_undegraded":"2024-03-12T03:22:34.284487+0000","last_fullsized":"2024-03-12T03:22:34.284487+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:35:00.013097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283707+0000","last_change":"2024-03-12T03:22:08.211172+0000","last_active":"2024-03-12T03:22:34.283707+0000","last_peered":"2024-03-12T03:22:34.283707+0000","last_clean":"2024-03-12T03:22:34.283707+0000","last_became_active":"2024-03-12T03:22:08.210751+0000","last_became_peered":"2024-03-12T03:22:08.210751+0000","last_unstale":"2024-03-12T03:22:34.283707+0000","last_undegraded":"2024-03-12T03:22:34.283707+0000","last_fullsized":"2024-03-12T03:22:34.283707+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:21.498847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282986+0000","last_change":"2024-03-12T03:22:04.197105+0000","last_active":"2024-03-12T03:22:34.282986+0000","last_peered":"2024-03-12T03:22:34.282986+0000","last_clean":"2024-03-12T03:22:34.282986+0000","last_became_active":"2024-03-12T03:22:04.196103+0000","last_became_peered":"2024-03-12T03:22:04.196103+0000","last_unstale":"2024-03-12T03:22:34.282986+0000","last_undegraded":"2024-03-12T03:22:34.282986+0000","last_fullsized":"2024-03-12T03:22:34.282986+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:18:01.851810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283243+0000","last_change":"2024-03-12T03:22:06.198533+0000","last_active":"2024-03-12T03:22:34.283243+0000","last_peered":"2024-03-12T03:22:34.283243+0000","last_clean":"2024-03-12T03:22:34.283243+0000","last_became_active":"2024-03-12T03:22:06.197935+0000","last_became_peered":"2024-03-12T03:22:06.197935+0000","last_unstale":"2024-03-12T03:22:34.283243+0000","last_undegraded":"2024-03-12T03:22:34.283243+0000","last_fullsized":"2024-03-12T03:22:34.283243+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:40:27.237249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283269+0000","last_change":"2024-03-12T03:22:10.224206+0000","last_active":"2024-03-12T03:22:34.283269+0000","last_peered":"2024-03-12T03:22:34.283269+0000","last_clean":"2024-03-12T03:22:34.283269+0000","last_became_active":"2024-03-12T03:22:10.223909+0000","last_became_peered":"2024-03-12T03:22:10.223909+0000","last_unstale":"2024-03-12T03:22:34.283269+0000","last_undegraded":"2024-03-12T03:22:34.283269+0000","last_fullsized":"2024-03-12T03:22:34.283269+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:29.616701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285130+0000","last_change":"2024-03-12T03:22:04.201316+0000","last_active":"2024-03-12T03:22:34.285130+0000","last_peered":"2024-03-12T03:22:34.285130+0000","last_clean":"2024-03-12T03:22:34.285130+0000","last_became_active":"2024-03-12T03:22:04.200809+0000","last_became_peered":"2024-03-12T03:22:04.200809+0000","last_unstale":"2024-03-12T03:22:34.285130+0000","last_undegraded":"2024-03-12T03:22:34.285130+0000","last_fullsized":"2024-03-12T03:22:34.285130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:52:51.411909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285781+0000","last_change":"2024-03-12T03:22:06.198420+0000","last_active":"2024-03-12T03:22:34.285781+0000","last_peered":"2024-03-12T03:22:34.285781+0000","last_clean":"2024-03-12T03:22:34.285781+0000","last_became_active":"2024-03-12T03:22:06.198089+0000","last_became_peered":"2024-03-12T03:22:06.198089+0000","last_unstale":"2024-03-12T03:22:34.285781+0000","last_undegraded":"2024-03-12T03:22:34.285781+0000","last_fullsized":"2024-03-12T03:22:34.285781+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:52:41.762099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:54.251965+0000","last_change":"2024-03-12T03:22:08.206318+0000","last_active":"2024-03-12T03:22:54.251965+0000","last_peered":"2024-03-12T03:22:54.251965+0000","last_clean":"2024-03-12T03:22:54.251965+0000","last_became_active":"2024-03-12T03:22:08.206039+0000","last_became_peered":"2024-03-12T03:22:08.206039+0000","last_unstale":"2024-03-12T03:22:54.251965+0000","last_undegraded":"2024-03-12T03:22:54.251965+0000","last_fullsized":"2024-03-12T03:22:54.251965+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:18:02.241476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.281904+0000","last_change":"2024-03-12T03:22:10.220045+0000","last_active":"2024-03-12T03:22:34.281904+0000","last_peered":"2024-03-12T03:22:34.281904+0000","last_clean":"2024-03-12T03:22:34.281904+0000","last_became_active":"2024-03-12T03:22:10.218364+0000","last_became_peered":"2024-03-12T03:22:10.218364+0000","last_unstale":"2024-03-12T03:22:34.281904+0000","last_undegraded":"2024-03-12T03:22:34.281904+0000","last_fullsized":"2024-03-12T03:22:34.281904+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:34:38.962346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283649+0000","last_change":"2024-03-12T03:22:04.195102+0000","last_active":"2024-03-12T03:22:34.283649+0000","last_peered":"2024-03-12T03:22:34.283649+0000","last_clean":"2024-03-12T03:22:34.283649+0000","last_became_active":"2024-03-12T03:22:04.194497+0000","last_became_peered":"2024-03-12T03:22:04.194497+0000","last_unstale":"2024-03-12T03:22:34.283649+0000","last_undegraded":"2024-03-12T03:22:34.283649+0000","last_fullsized":"2024-03-12T03:22:34.283649+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:34:38.035219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:54.251845+0000","last_change":"2024-03-12T03:22:06.201933+0000","last_active":"2024-03-12T03:22:54.251845+0000","last_peered":"2024-03-12T03:22:54.251845+0000","last_clean":"2024-03-12T03:22:54.251845+0000","last_became_active":"2024-03-12T03:22:06.201325+0000","last_became_peered":"2024-03-12T03:22:06.201325+0000","last_unstale":"2024-03-12T03:22:54.251845+0000","last_undegraded":"2024-03-12T03:22:54.251845+0000","last_fullsized":"2024-03-12T03:22:54.251845+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:46:08.259243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283198+0000","last_change":"2024-03-12T03:22:08.207299+0000","last_active":"2024-03-12T03:22:34.283198+0000","last_peered":"2024-03-12T03:22:34.283198+0000","last_clean":"2024-03-12T03:22:34.283198+0000","last_became_active":"2024-03-12T03:22:08.206458+0000","last_became_peered":"2024-03-12T03:22:08.206458+0000","last_unstale":"2024-03-12T03:22:34.283198+0000","last_undegraded":"2024-03-12T03:22:34.283198+0000","last_fullsized":"2024-03-12T03:22:34.283198+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:24:07.066415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282763+0000","last_change":"2024-03-12T03:22:10.226362+0000","last_active":"2024-03-12T03:22:34.282763+0000","last_peered":"2024-03-12T03:22:34.282763+0000","last_clean":"2024-03-12T03:22:34.282763+0000","last_became_active":"2024-03-12T03:22:10.225984+0000","last_became_peered":"2024-03-12T03:22:10.225984+0000","last_unstale":"2024-03-12T03:22:34.282763+0000","last_undegraded":"2024-03-12T03:22:34.282763+0000","last_fullsized":"2024-03-12T03:22:34.282763+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:34:05.731040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284032+0000","last_change":"2024-03-12T03:22:04.196821+0000","last_active":"2024-03-12T03:22:34.284032+0000","last_peered":"2024-03-12T03:22:34.284032+0000","last_clean":"2024-03-12T03:22:34.284032+0000","last_became_active":"2024-03-12T03:22:04.194671+0000","last_became_peered":"2024-03-12T03:22:04.194671+0000","last_unstale":"2024-03-12T03:22:34.284032+0000","last_undegraded":"2024-03-12T03:22:34.284032+0000","last_fullsized":"2024-03-12T03:22:34.284032+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:37:39.786305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'25","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284718+0000","last_change":"2024-03-12T03:22:06.201043+0000","last_active":"2024-03-12T03:22:34.284718+0000","last_peered":"2024-03-12T03:22:34.284718+0000","last_clean":"2024-03-12T03:22:34.284718+0000","last_became_active":"2024-03-12T03:22:06.200503+0000","last_became_peered":"2024-03-12T03:22:06.200503+0000","last_unstale":"2024-03-12T03:22:34.284718+0000","last_undegraded":"2024-03-12T03:22:34.284718+0000","last_fullsized":"2024-03-12T03:22:34.284718+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:02:43.666211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282778+0000","last_change":"2024-03-12T03:22:08.210798+0000","last_active":"2024-03-12T03:22:34.282778+0000","last_peered":"2024-03-12T03:22:34.282778+0000","last_clean":"2024-03-12T03:22:34.282778+0000","last_became_active":"2024-03-12T03:22:08.210432+0000","last_became_peered":"2024-03-12T03:22:08.210432+0000","last_unstale":"2024-03-12T03:22:34.282778+0000","last_undegraded":"2024-03-12T03:22:34.282778+0000","last_fullsized":"2024-03-12T03:22:34.282778+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:14:18.597046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284368+0000","last_change":"2024-03-12T03:22:10.225026+0000","last_active":"2024-03-12T03:22:34.284368+0000","last_peered":"2024-03-12T03:22:34.284368+0000","last_clean":"2024-03-12T03:22:34.284368+0000","last_became_active":"2024-03-12T03:22:10.224555+0000","last_became_peered":"2024-03-12T03:22:10.224555+0000","last_unstale":"2024-03-12T03:22:34.284368+0000","last_undegraded":"2024-03-12T03:22:34.284368+0000","last_fullsized":"2024-03-12T03:22:34.284368+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:38:46.416155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'30","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282615+0000","last_change":"2024-03-12T03:22:06.199102+0000","last_active":"2024-03-12T03:22:34.282615+0000","last_peered":"2024-03-12T03:22:34.282615+0000","last_clean":"2024-03-12T03:22:34.282615+0000","last_became_active":"2024-03-12T03:22:06.198813+0000","last_became_peered":"2024-03-12T03:22:06.198813+0000","last_unstale":"2024-03-12T03:22:34.282615+0000","last_undegraded":"2024-03-12T03:22:34.282615+0000","last_fullsized":"2024-03-12T03:22:34.282615+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:43:43.674860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282463+0000","last_change":"2024-03-12T03:22:04.195312+0000","last_active":"2024-03-12T03:22:34.282463+0000","last_peered":"2024-03-12T03:22:34.282463+0000","last_clean":"2024-03-12T03:22:34.282463+0000","last_became_active":"2024-03-12T03:22:04.194879+0000","last_became_peered":"2024-03-12T03:22:04.194879+0000","last_unstale":"2024-03-12T03:22:34.282463+0000","last_undegraded":"2024-03-12T03:22:34.282463+0000","last_fullsized":"2024-03-12T03:22:34.282463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:24:35.264530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284963+0000","last_change":"2024-03-12T03:22:08.211589+0000","last_active":"2024-03-12T03:22:34.284963+0000","last_peered":"2024-03-12T03:22:34.284963+0000","last_clean":"2024-03-12T03:22:34.284963+0000","last_became_active":"2024-03-12T03:22:08.210985+0000","last_became_peered":"2024-03-12T03:22:08.210985+0000","last_unstale":"2024-03-12T03:22:34.284963+0000","last_undegraded":"2024-03-12T03:22:34.284963+0000","last_fullsized":"2024-03-12T03:22:34.284963+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:25:00.994271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283556+0000","last_change":"2024-03-12T03:22:10.221345+0000","last_active":"2024-03-12T03:22:34.283556+0000","last_peered":"2024-03-12T03:22:34.283556+0000","last_clean":"2024-03-12T03:22:34.283556+0000","last_became_active":"2024-03-12T03:22:10.221037+0000","last_became_peered":"2024-03-12T03:22:10.221037+0000","last_unstale":"2024-03-12T03:22:34.283556+0000","last_undegraded":"2024-03-12T03:22:34.283556+0000","last_fullsized":"2024-03-12T03:22:34.283556+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:34:37.475961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282079+0000","last_change":"2024-03-12T03:22:06.197744+0000","last_active":"2024-03-12T03:22:34.282079+0000","last_peered":"2024-03-12T03:22:34.282079+0000","last_clean":"2024-03-12T03:22:34.282079+0000","last_became_active":"2024-03-12T03:22:06.197400+0000","last_became_peered":"2024-03-12T03:22:06.197400+0000","last_unstale":"2024-03-12T03:22:34.282079+0000","last_undegraded":"2024-03-12T03:22:34.282079+0000","last_fullsized":"2024-03-12T03:22:34.282079+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:24:40.716729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284226+0000","last_change":"2024-03-12T03:22:04.196599+0000","last_active":"2024-03-12T03:22:34.284226+0000","last_peered":"2024-03-12T03:22:34.284226+0000","last_clean":"2024-03-12T03:22:34.284226+0000","last_became_active":"2024-03-12T03:22:04.195845+0000","last_became_peered":"2024-03-12T03:22:04.195845+0000","last_unstale":"2024-03-12T03:22:34.284226+0000","last_undegraded":"2024-03-12T03:22:34.284226+0000","last_fullsized":"2024-03-12T03:22:34.284226+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:02:00.399884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:36.384612+0000","last_change":"2024-03-12T03:21:49.582258+0000","last_active":"2024-03-12T03:22:36.384612+0000","last_peered":"2024-03-12T03:22:36.384612+0000","last_clean":"2024-03-12T03:22:36.384612+0000","last_became_active":"2024-03-12T03:21:49.581839+0000","last_became_peered":"2024-03-12T03:21:49.581839+0000","last_unstale":"2024-03-12T03:22:36.384612+0000","last_undegraded":"2024-03-12T03:22:36.384612+0000","last_fullsized":"2024-03-12T03:22:36.384612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:21:48.573477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:21:48.573477+0000","last_clean_scrub_stamp":"2024-03-12T03:21:48.573477+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:07:19.213260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284709+0000","last_change":"2024-03-12T03:22:08.211294+0000","last_active":"2024-03-12T03:22:34.284709+0000","last_peered":"2024-03-12T03:22:34.284709+0000","last_clean":"2024-03-12T03:22:34.284709+0000","last_became_active":"2024-03-12T03:22:08.210941+0000","last_became_peered":"2024-03-12T03:22:08.210941+0000","last_unstale":"2024-03-12T03:22:34.284709+0000","last_undegraded":"2024-03-12T03:22:34.284709+0000","last_fullsized":"2024-03-12T03:22:34.284709+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:53:45.896536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283918+0000","last_change":"2024-03-12T03:22:10.220793+0000","last_active":"2024-03-12T03:22:34.283918+0000","last_peered":"2024-03-12T03:22:34.283918+0000","last_clean":"2024-03-12T03:22:34.283918+0000","last_became_active":"2024-03-12T03:22:10.220192+0000","last_became_peered":"2024-03-12T03:22:10.220192+0000","last_unstale":"2024-03-12T03:22:34.283918+0000","last_undegraded":"2024-03-12T03:22:34.283918+0000","last_fullsized":"2024-03-12T03:22:34.283918+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:03:05.188945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'20","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282852+0000","last_change":"2024-03-12T03:22:06.198127+0000","last_active":"2024-03-12T03:22:34.282852+0000","last_peered":"2024-03-12T03:22:34.282852+0000","last_clean":"2024-03-12T03:22:34.282852+0000","last_became_active":"2024-03-12T03:22:06.197617+0000","last_became_peered":"2024-03-12T03:22:06.197617+0000","last_unstale":"2024-03-12T03:22:34.282852+0000","last_undegraded":"2024-03-12T03:22:34.282852+0000","last_fullsized":"2024-03-12T03:22:34.282852+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:18.174161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283164+0000","last_change":"2024-03-12T03:22:04.198749+0000","last_active":"2024-03-12T03:22:34.283164+0000","last_peered":"2024-03-12T03:22:34.283164+0000","last_clean":"2024-03-12T03:22:34.283164+0000","last_became_active":"2024-03-12T03:22:04.197328+0000","last_became_peered":"2024-03-12T03:22:04.197328+0000","last_unstale":"2024-03-12T03:22:34.283164+0000","last_undegraded":"2024-03-12T03:22:34.283164+0000","last_fullsized":"2024-03-12T03:22:34.283164+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:25:06.158061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284719+0000","last_change":"2024-03-12T03:22:08.206070+0000","last_active":"2024-03-12T03:22:34.284719+0000","last_peered":"2024-03-12T03:22:34.284719+0000","last_clean":"2024-03-12T03:22:34.284719+0000","last_became_active":"2024-03-12T03:22:08.205678+0000","last_became_peered":"2024-03-12T03:22:08.205678+0000","last_unstale":"2024-03-12T03:22:34.284719+0000","last_undegraded":"2024-03-12T03:22:34.284719+0000","last_fullsized":"2024-03-12T03:22:34.284719+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:47:40.986036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283683+0000","last_change":"2024-03-12T03:22:10.220474+0000","last_active":"2024-03-12T03:22:34.283683+0000","last_peered":"2024-03-12T03:22:34.283683+0000","last_clean":"2024-03-12T03:22:34.283683+0000","last_became_active":"2024-03-12T03:22:10.219971+0000","last_became_peered":"2024-03-12T03:22:10.219971+0000","last_unstale":"2024-03-12T03:22:34.283683+0000","last_undegraded":"2024-03-12T03:22:34.283683+0000","last_fullsized":"2024-03-12T03:22:34.283683+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:12:32.284434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285798+0000","last_change":"2024-03-12T03:22:04.201436+0000","last_active":"2024-03-12T03:22:34.285798+0000","last_peered":"2024-03-12T03:22:34.285798+0000","last_clean":"2024-03-12T03:22:34.285798+0000","last_became_active":"2024-03-12T03:22:04.200929+0000","last_became_peered":"2024-03-12T03:22:04.200929+0000","last_unstale":"2024-03-12T03:22:34.285798+0000","last_undegraded":"2024-03-12T03:22:34.285798+0000","last_fullsized":"2024-03-12T03:22:34.285798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:29:32.733354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'34","reported_seq":95,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:54.251752+0000","last_change":"2024-03-12T03:22:06.199637+0000","last_active":"2024-03-12T03:22:54.251752+0000","last_peered":"2024-03-12T03:22:54.251752+0000","last_clean":"2024-03-12T03:22:54.251752+0000","last_became_active":"2024-03-12T03:22:06.199323+0000","last_became_peered":"2024-03-12T03:22:06.199323+0000","last_unstale":"2024-03-12T03:22:54.251752+0000","last_undegraded":"2024-03-12T03:22:54.251752+0000","last_fullsized":"2024-03-12T03:22:54.251752+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:41:26.795590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:54.251925+0000","last_change":"2024-03-12T03:22:08.207528+0000","last_active":"2024-03-12T03:22:54.251925+0000","last_peered":"2024-03-12T03:22:54.251925+0000","last_clean":"2024-03-12T03:22:54.251925+0000","last_became_active":"2024-03-12T03:22:08.207232+0000","last_became_peered":"2024-03-12T03:22:08.207232+0000","last_unstale":"2024-03-12T03:22:54.251925+0000","last_undegraded":"2024-03-12T03:22:54.251925+0000","last_fullsized":"2024-03-12T03:22:54.251925+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:53.361024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283035+0000","last_change":"2024-03-12T03:22:10.224665+0000","last_active":"2024-03-12T03:22:34.283035+0000","last_peered":"2024-03-12T03:22:34.283035+0000","last_clean":"2024-03-12T03:22:34.283035+0000","last_became_active":"2024-03-12T03:22:10.223400+0000","last_became_peered":"2024-03-12T03:22:10.223400+0000","last_unstale":"2024-03-12T03:22:34.283035+0000","last_undegraded":"2024-03-12T03:22:34.283035+0000","last_fullsized":"2024-03-12T03:22:34.283035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:55:17.460790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284316+0000","last_change":"2024-03-12T03:22:04.194555+0000","last_active":"2024-03-12T03:22:34.284316+0000","last_peered":"2024-03-12T03:22:34.284316+0000","last_clean":"2024-03-12T03:22:34.284316+0000","last_became_active":"2024-03-12T03:22:04.193871+0000","last_became_peered":"2024-03-12T03:22:04.193871+0000","last_unstale":"2024-03-12T03:22:34.284316+0000","last_undegraded":"2024-03-12T03:22:34.284316+0000","last_fullsized":"2024-03-12T03:22:34.284316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:06:08.226618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'19","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285571+0000","last_change":"2024-03-12T03:22:06.201091+0000","last_active":"2024-03-12T03:22:34.285571+0000","last_peered":"2024-03-12T03:22:34.285571+0000","last_clean":"2024-03-12T03:22:34.285571+0000","last_became_active":"2024-03-12T03:22:06.200716+0000","last_became_peered":"2024-03-12T03:22:06.200716+0000","last_unstale":"2024-03-12T03:22:34.285571+0000","last_undegraded":"2024-03-12T03:22:34.285571+0000","last_fullsized":"2024-03-12T03:22:34.285571+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:14:56.705991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:54.251680+0000","last_change":"2024-03-12T03:22:08.209196+0000","last_active":"2024-03-12T03:22:54.251680+0000","last_peered":"2024-03-12T03:22:54.251680+0000","last_clean":"2024-03-12T03:22:54.251680+0000","last_became_active":"2024-03-12T03:22:08.208819+0000","last_became_peered":"2024-03-12T03:22:08.208819+0000","last_unstale":"2024-03-12T03:22:54.251680+0000","last_undegraded":"2024-03-12T03:22:54.251680+0000","last_fullsized":"2024-03-12T03:22:54.251680+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:19.930661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283262+0000","last_change":"2024-03-12T03:22:10.224180+0000","last_active":"2024-03-12T03:22:34.283262+0000","last_peered":"2024-03-12T03:22:34.283262+0000","last_clean":"2024-03-12T03:22:34.283262+0000","last_became_active":"2024-03-12T03:22:10.223890+0000","last_became_peered":"2024-03-12T03:22:10.223890+0000","last_unstale":"2024-03-12T03:22:34.283262+0000","last_undegraded":"2024-03-12T03:22:34.283262+0000","last_fullsized":"2024-03-12T03:22:34.283262+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:33:28.811259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284751+0000","last_change":"2024-03-12T03:22:04.197435+0000","last_active":"2024-03-12T03:22:34.284751+0000","last_peered":"2024-03-12T03:22:34.284751+0000","last_clean":"2024-03-12T03:22:34.284751+0000","last_became_active":"2024-03-12T03:22:04.196633+0000","last_became_peered":"2024-03-12T03:22:04.196633+0000","last_unstale":"2024-03-12T03:22:34.284751+0000","last_undegraded":"2024-03-12T03:22:34.284751+0000","last_fullsized":"2024-03-12T03:22:34.284751+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:15:16.765568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283860+0000","last_change":"2024-03-12T03:22:06.198330+0000","last_active":"2024-03-12T03:22:34.283860+0000","last_peered":"2024-03-12T03:22:34.283860+0000","last_clean":"2024-03-12T03:22:34.283860+0000","last_became_active":"2024-03-12T03:22:06.197900+0000","last_became_peered":"2024-03-12T03:22:06.197900+0000","last_unstale":"2024-03-12T03:22:34.283860+0000","last_undegraded":"2024-03-12T03:22:34.283860+0000","last_fullsized":"2024-03-12T03:22:34.283860+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:02:20.974220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282992+0000","last_change":"2024-03-12T03:22:08.206148+0000","last_active":"2024-03-12T03:22:34.282992+0000","last_peered":"2024-03-12T03:22:34.282992+0000","last_clean":"2024-03-12T03:22:34.282992+0000","last_became_active":"2024-03-12T03:22:08.205652+0000","last_became_peered":"2024-03-12T03:22:08.205652+0000","last_unstale":"2024-03-12T03:22:34.282992+0000","last_undegraded":"2024-03-12T03:22:34.282992+0000","last_fullsized":"2024-03-12T03:22:34.282992+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:10:44.789210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284152+0000","last_change":"2024-03-12T03:22:10.220570+0000","last_active":"2024-03-12T03:22:34.284152+0000","last_peered":"2024-03-12T03:22:34.284152+0000","last_clean":"2024-03-12T03:22:34.284152+0000","last_became_active":"2024-03-12T03:22:10.219865+0000","last_became_peered":"2024-03-12T03:22:10.219865+0000","last_unstale":"2024-03-12T03:22:34.284152+0000","last_undegraded":"2024-03-12T03:22:34.284152+0000","last_fullsized":"2024-03-12T03:22:34.284152+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:06:01.434470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:54.251821+0000","last_change":"2024-03-12T03:22:08.212329+0000","last_active":"2024-03-12T03:22:54.251821+0000","last_peered":"2024-03-12T03:22:54.251821+0000","last_clean":"2024-03-12T03:22:54.251821+0000","last_became_active":"2024-03-12T03:22:08.211998+0000","last_became_peered":"2024-03-12T03:22:08.211998+0000","last_unstale":"2024-03-12T03:22:54.251821+0000","last_undegraded":"2024-03-12T03:22:54.251821+0000","last_fullsized":"2024-03-12T03:22:54.251821+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:39:58.131467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283267+0000","last_change":"2024-03-12T03:22:04.198742+0000","last_active":"2024-03-12T03:22:34.283267+0000","last_peered":"2024-03-12T03:22:34.283267+0000","last_clean":"2024-03-12T03:22:34.283267+0000","last_became_active":"2024-03-12T03:22:04.197470+0000","last_became_peered":"2024-03-12T03:22:04.197470+0000","last_unstale":"2024-03-12T03:22:34.283267+0000","last_undegraded":"2024-03-12T03:22:34.283267+0000","last_fullsized":"2024-03-12T03:22:34.283267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:19:18.414259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'24","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283572+0000","last_change":"2024-03-12T03:22:06.198364+0000","last_active":"2024-03-12T03:22:34.283572+0000","last_peered":"2024-03-12T03:22:34.283572+0000","last_clean":"2024-03-12T03:22:34.283572+0000","last_became_active":"2024-03-12T03:22:06.198035+0000","last_became_peered":"2024-03-12T03:22:06.198035+0000","last_unstale":"2024-03-12T03:22:34.283572+0000","last_undegraded":"2024-03-12T03:22:34.283572+0000","last_fullsized":"2024-03-12T03:22:34.283572+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:37:25.895897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284881+0000","last_change":"2024-03-12T03:22:10.224148+0000","last_active":"2024-03-12T03:22:34.284881+0000","last_peered":"2024-03-12T03:22:34.284881+0000","last_clean":"2024-03-12T03:22:34.284881+0000","last_became_active":"2024-03-12T03:22:10.223838+0000","last_became_peered":"2024-03-12T03:22:10.223838+0000","last_unstale":"2024-03-12T03:22:34.284881+0000","last_undegraded":"2024-03-12T03:22:34.284881+0000","last_fullsized":"2024-03-12T03:22:34.284881+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:50:49.806487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284871+0000","last_change":"2024-03-12T03:22:08.211587+0000","last_active":"2024-03-12T03:22:34.284871+0000","last_peered":"2024-03-12T03:22:34.284871+0000","last_clean":"2024-03-12T03:22:34.284871+0000","last_became_active":"2024-03-12T03:22:08.210504+0000","last_became_peered":"2024-03-12T03:22:08.210504+0000","last_unstale":"2024-03-12T03:22:34.284871+0000","last_undegraded":"2024-03-12T03:22:34.284871+0000","last_fullsized":"2024-03-12T03:22:34.284871+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:15.683378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282565+0000","last_change":"2024-03-12T03:22:04.195035+0000","last_active":"2024-03-12T03:22:34.282565+0000","last_peered":"2024-03-12T03:22:34.282565+0000","last_clean":"2024-03-12T03:22:34.282565+0000","last_became_active":"2024-03-12T03:22:04.194004+0000","last_became_peered":"2024-03-12T03:22:04.194004+0000","last_unstale":"2024-03-12T03:22:34.282565+0000","last_undegraded":"2024-03-12T03:22:34.282565+0000","last_fullsized":"2024-03-12T03:22:34.282565+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:27:48.149760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'27","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283059+0000","last_change":"2024-03-12T03:22:06.199995+0000","last_active":"2024-03-12T03:22:34.283059+0000","last_peered":"2024-03-12T03:22:34.283059+0000","last_clean":"2024-03-12T03:22:34.283059+0000","last_became_active":"2024-03-12T03:22:06.198766+0000","last_became_peered":"2024-03-12T03:22:06.198766+0000","last_unstale":"2024-03-12T03:22:34.283059+0000","last_undegraded":"2024-03-12T03:22:34.283059+0000","last_fullsized":"2024-03-12T03:22:34.283059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:46:44.985314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284697+0000","last_change":"2024-03-12T03:22:10.219335+0000","last_active":"2024-03-12T03:22:34.284697+0000","last_peered":"2024-03-12T03:22:34.284697+0000","last_clean":"2024-03-12T03:22:34.284697+0000","last_became_active":"2024-03-12T03:22:10.219119+0000","last_became_peered":"2024-03-12T03:22:10.219119+0000","last_unstale":"2024-03-12T03:22:34.284697+0000","last_undegraded":"2024-03-12T03:22:34.284697+0000","last_fullsized":"2024-03-12T03:22:34.284697+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:11:00.361204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285148+0000","last_change":"2024-03-12T03:22:04.196305+0000","last_active":"2024-03-12T03:22:34.285148+0000","last_peered":"2024-03-12T03:22:34.285148+0000","last_clean":"2024-03-12T03:22:34.285148+0000","last_became_active":"2024-03-12T03:22:04.195909+0000","last_became_peered":"2024-03-12T03:22:04.195909+0000","last_unstale":"2024-03-12T03:22:34.285148+0000","last_undegraded":"2024-03-12T03:22:34.285148+0000","last_fullsized":"2024-03-12T03:22:34.285148+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:04:28.257235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282266+0000","last_change":"2024-03-12T03:22:06.198321+0000","last_active":"2024-03-12T03:22:34.282266+0000","last_peered":"2024-03-12T03:22:34.282266+0000","last_clean":"2024-03-12T03:22:34.282266+0000","last_became_active":"2024-03-12T03:22:06.197900+0000","last_became_peered":"2024-03-12T03:22:06.197900+0000","last_unstale":"2024-03-12T03:22:34.282266+0000","last_undegraded":"2024-03-12T03:22:34.282266+0000","last_fullsized":"2024-03-12T03:22:34.282266+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:55:20.488421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284288+0000","last_change":"2024-03-12T03:22:08.211995+0000","last_active":"2024-03-12T03:22:34.284288+0000","last_peered":"2024-03-12T03:22:34.284288+0000","last_clean":"2024-03-12T03:22:34.284288+0000","last_became_active":"2024-03-12T03:22:08.211455+0000","last_became_peered":"2024-03-12T03:22:08.211455+0000","last_unstale":"2024-03-12T03:22:34.284288+0000","last_undegraded":"2024-03-12T03:22:34.284288+0000","last_fullsized":"2024-03-12T03:22:34.284288+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:15:54.087061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282664+0000","last_change":"2024-03-12T03:22:10.226502+0000","last_active":"2024-03-12T03:22:34.282664+0000","last_peered":"2024-03-12T03:22:34.282664+0000","last_clean":"2024-03-12T03:22:34.282664+0000","last_became_active":"2024-03-12T03:22:10.226176+0000","last_became_peered":"2024-03-12T03:22:10.226176+0000","last_unstale":"2024-03-12T03:22:34.282664+0000","last_undegraded":"2024-03-12T03:22:34.282664+0000","last_fullsized":"2024-03-12T03:22:34.282664+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:25:14.324426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:54.251898+0000","last_change":"2024-03-12T03:22:08.211510+0000","last_active":"2024-03-12T03:22:54.251898+0000","last_peered":"2024-03-12T03:22:54.251898+0000","last_clean":"2024-03-12T03:22:54.251898+0000","last_became_active":"2024-03-12T03:22:08.211148+0000","last_became_peered":"2024-03-12T03:22:08.211148+0000","last_unstale":"2024-03-12T03:22:54.251898+0000","last_undegraded":"2024-03-12T03:22:54.251898+0000","last_fullsized":"2024-03-12T03:22:54.251898+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:13:00.298813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284460+0000","last_change":"2024-03-12T03:22:04.196934+0000","last_active":"2024-03-12T03:22:34.284460+0000","last_peered":"2024-03-12T03:22:34.284460+0000","last_clean":"2024-03-12T03:22:34.284460+0000","last_became_active":"2024-03-12T03:22:04.196138+0000","last_became_peered":"2024-03-12T03:22:04.196138+0000","last_unstale":"2024-03-12T03:22:34.284460+0000","last_undegraded":"2024-03-12T03:22:34.284460+0000","last_fullsized":"2024-03-12T03:22:34.284460+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:11:29.886706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'25","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284151+0000","last_change":"2024-03-12T03:22:06.201126+0000","last_active":"2024-03-12T03:22:34.284151+0000","last_peered":"2024-03-12T03:22:34.284151+0000","last_clean":"2024-03-12T03:22:34.284151+0000","last_became_active":"2024-03-12T03:22:06.200623+0000","last_became_peered":"2024-03-12T03:22:06.200623+0000","last_unstale":"2024-03-12T03:22:34.284151+0000","last_undegraded":"2024-03-12T03:22:34.284151+0000","last_fullsized":"2024-03-12T03:22:34.284151+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:43:02.117846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283724+0000","last_change":"2024-03-12T03:22:10.219473+0000","last_active":"2024-03-12T03:22:34.283724+0000","last_peered":"2024-03-12T03:22:34.283724+0000","last_clean":"2024-03-12T03:22:34.283724+0000","last_became_active":"2024-03-12T03:22:10.219232+0000","last_became_peered":"2024-03-12T03:22:10.219232+0000","last_unstale":"2024-03-12T03:22:34.283724+0000","last_undegraded":"2024-03-12T03:22:34.283724+0000","last_fullsized":"2024-03-12T03:22:34.283724+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:06:34.371955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284428+0000","last_change":"2024-03-12T03:22:04.197232+0000","last_active":"2024-03-12T03:22:34.284428+0000","last_peered":"2024-03-12T03:22:34.284428+0000","last_clean":"2024-03-12T03:22:34.284428+0000","last_became_active":"2024-03-12T03:22:04.196967+0000","last_became_peered":"2024-03-12T03:22:04.196967+0000","last_unstale":"2024-03-12T03:22:34.284428+0000","last_undegraded":"2024-03-12T03:22:34.284428+0000","last_fullsized":"2024-03-12T03:22:34.284428+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:17:47.670181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285931+0000","last_change":"2024-03-12T03:22:06.201892+0000","last_active":"2024-03-12T03:22:34.285931+0000","last_peered":"2024-03-12T03:22:34.285931+0000","last_clean":"2024-03-12T03:22:34.285931+0000","last_became_active":"2024-03-12T03:22:06.201464+0000","last_became_peered":"2024-03-12T03:22:06.201464+0000","last_unstale":"2024-03-12T03:22:34.285931+0000","last_undegraded":"2024-03-12T03:22:34.285931+0000","last_fullsized":"2024-03-12T03:22:34.285931+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:47:15.357161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283520+0000","last_change":"2024-03-12T03:22:08.213520+0000","last_active":"2024-03-12T03:22:34.283520+0000","last_peered":"2024-03-12T03:22:34.283520+0000","last_clean":"2024-03-12T03:22:34.283520+0000","last_became_active":"2024-03-12T03:22:08.213170+0000","last_became_peered":"2024-03-12T03:22:08.213170+0000","last_unstale":"2024-03-12T03:22:34.283520+0000","last_undegraded":"2024-03-12T03:22:34.283520+0000","last_fullsized":"2024-03-12T03:22:34.283520+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:18:14.436804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.281686+0000","last_change":"2024-03-12T03:22:10.220245+0000","last_active":"2024-03-12T03:22:34.281686+0000","last_peered":"2024-03-12T03:22:34.281686+0000","last_clean":"2024-03-12T03:22:34.281686+0000","last_became_active":"2024-03-12T03:22:10.218325+0000","last_became_peered":"2024-03-12T03:22:10.218325+0000","last_unstale":"2024-03-12T03:22:34.281686+0000","last_undegraded":"2024-03-12T03:22:34.281686+0000","last_fullsized":"2024-03-12T03:22:34.281686+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:27:45.735368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285062+0000","last_change":"2024-03-12T03:22:04.201634+0000","last_active":"2024-03-12T03:22:34.285062+0000","last_peered":"2024-03-12T03:22:34.285062+0000","last_clean":"2024-03-12T03:22:34.285062+0000","last_became_active":"2024-03-12T03:22:04.200975+0000","last_became_peered":"2024-03-12T03:22:04.200975+0000","last_unstale":"2024-03-12T03:22:34.285062+0000","last_undegraded":"2024-03-12T03:22:34.285062+0000","last_fullsized":"2024-03-12T03:22:34.285062+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:56:15.238174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.285697+0000","last_change":"2024-03-12T03:22:06.203022+0000","last_active":"2024-03-12T03:22:34.285697+0000","last_peered":"2024-03-12T03:22:34.285697+0000","last_clean":"2024-03-12T03:22:34.285697+0000","last_became_active":"2024-03-12T03:22:06.202692+0000","last_became_peered":"2024-03-12T03:22:06.202692+0000","last_unstale":"2024-03-12T03:22:34.285697+0000","last_undegraded":"2024-03-12T03:22:34.285697+0000","last_fullsized":"2024-03-12T03:22:34.285697+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:28:41.561273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.282098+0000","last_change":"2024-03-12T03:22:08.209584+0000","last_active":"2024-03-12T03:22:34.282098+0000","last_peered":"2024-03-12T03:22:34.282098+0000","last_clean":"2024-03-12T03:22:34.282098+0000","last_became_active":"2024-03-12T03:22:08.209232+0000","last_became_peered":"2024-03-12T03:22:08.209232+0000","last_unstale":"2024-03-12T03:22:34.282098+0000","last_undegraded":"2024-03-12T03:22:34.282098+0000","last_fullsized":"2024-03-12T03:22:34.282098+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:14:44.071919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283165+0000","last_change":"2024-03-12T03:22:10.222550+0000","last_active":"2024-03-12T03:22:34.283165+0000","last_peered":"2024-03-12T03:22:34.283165+0000","last_clean":"2024-03-12T03:22:34.283165+0000","last_became_active":"2024-03-12T03:22:10.221438+0000","last_became_peered":"2024-03-12T03:22:10.221438+0000","last_unstale":"2024-03-12T03:22:34.283165+0000","last_undegraded":"2024-03-12T03:22:34.283165+0000","last_fullsized":"2024-03-12T03:22:34.283165+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:09.205073+0000","last_clean_scrub_stamp":"2024-03-12T03:22:09.205073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:46:20.811352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284148+0000","last_change":"2024-03-12T03:22:04.194652+0000","last_active":"2024-03-12T03:22:34.284148+0000","last_peered":"2024-03-12T03:22:34.284148+0000","last_clean":"2024-03-12T03:22:34.284148+0000","last_became_active":"2024-03-12T03:22:04.193908+0000","last_became_peered":"2024-03-12T03:22:04.193908+0000","last_unstale":"2024-03-12T03:22:34.284148+0000","last_undegraded":"2024-03-12T03:22:34.284148+0000","last_fullsized":"2024-03-12T03:22:34.284148+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:03.180597+0000","last_clean_scrub_stamp":"2024-03-12T03:22:03.180597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:18:08.389201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.284854+0000","last_change":"2024-03-12T03:22:06.203115+0000","last_active":"2024-03-12T03:22:34.284854+0000","last_peered":"2024-03-12T03:22:34.284854+0000","last_clean":"2024-03-12T03:22:34.284854+0000","last_became_active":"2024-03-12T03:22:06.202656+0000","last_became_peered":"2024-03-12T03:22:06.202656+0000","last_unstale":"2024-03-12T03:22:34.284854+0000","last_undegraded":"2024-03-12T03:22:34.284854+0000","last_fullsized":"2024-03-12T03:22:34.284854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:05.185984+0000","last_clean_scrub_stamp":"2024-03-12T03:22:05.185984+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:26:11.403057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-12T03:22:34.283163+0000","last_change":"2024-03-12T03:22:08.206158+0000","last_active":"2024-03-12T03:22:34.283163+0000","last_peered":"2024-03-12T03:22:34.283163+0000","last_clean":"2024-03-12T03:22:34.283163+0000","last_became_active":"2024-03-12T03:22:08.205889+0000","last_became_peered":"2024-03-12T03:22:08.205889+0000","last_unstale":"2024-03-12T03:22:34.283163+0000","last_undegraded":"2024-03-12T03:22:34.283163+0000","last_fullsized":"2024-03-12T03:22:34.283163+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:22:07.193725+0000","last_clean_scrub_stamp":"2024-03-12T03:22:07.193725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:35:06.325638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":587,"ondisk_log_size":587,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411345,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93704692,"statfs":{"total":95990841344,"available":95953604608,"internally_reserved":0,"allocated":1052672,"data_stored":700191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738396,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93704692,"statfs":{"total":95990841344,"available":95953604608,"internally_reserved":0,"allocated":1052672,"data_stored":700191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:23:04.657 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-12T03:23:04.657 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-12T03:23:04.658 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-12T03:23:04.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-12T03:23:04.914 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:23:05.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:05 smithi007 ceph-mon[25800]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:23:05.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:05 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:05.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:05 smithi170 ceph-mon[27708]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:23:05.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:05 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:05.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:05 smithi033 ceph-mon[27552]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:23:05.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:05 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:05.968 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:05 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: ::ffff:172.21.15.33 - - [12/Mar/2024:03:23:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:23:05.968 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:23:05 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a[48326]: ts=2024-03-12T03:23:05.748Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003183685s 2024-03-12T03:23:06.268 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:06 smithi007 ceph-mon[25800]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:06.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:06 smithi170 ceph-mon[27708]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:06.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:06 smithi033 ceph-mon[27552]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:06.733 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-12T03:23:06.733 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-12T03:23:07.348 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:07 smithi007 ceph-mon[25800]: from='client.? 172.21.15.7:0/24999205' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T03:23:07.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:07 smithi170 ceph-mon[27708]: from='client.? 172.21.15.7:0/24999205' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T03:23:07.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:07 smithi033 ceph-mon[27552]: from='client.? 172.21.15.7:0/24999205' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T03:23:07.689 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-12T03:23:07.689 INFO:tasks.cephadm:Setup complete, yielding 2024-03-12T03:23:07.690 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T03:23:07.700 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-03-12T03:23:07.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-12T03:23:07.944 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:23:08.311 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:08 smithi007 ceph-mon[25800]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:08.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:08 smithi170 ceph-mon[27708]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:08.402 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:08 smithi033 ceph-mon[27552]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:09.559 INFO:teuthology.orchestra.run.smithi007.stdout:167 167 2024-03-12T03:23:10.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-12T03:23:10.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:10 smithi170 ceph-mon[27708]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:10.391 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:10 smithi033 ceph-mon[27552]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:10.418 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:23:10.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:10 smithi007 ceph-mon[25800]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:11.013 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:10 smithi033 systemd[1]: Starting Ceph grafana.a for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:23:11.345 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 podman[37868]: 2024-03-12 03:23:10.967951913 +0000 UTC m=+0.017087097 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-12T03:23:11.345 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 podman[37868]: 2024-03-12 03:23:11.083496709 +0000 UTC m=+0.132631929 container create a3c1de778e0477eda318c177b977e9ad97ff7a8fbb0dd2e799516c21d822124b (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a, release=239.1651231664, io.openshift.expose-services=, version=8.5, architecture=x86_64, io.buildah.version=1.32.0, name=ubi8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897) 2024-03-12T03:23:11.345 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 podman[37868]: 2024-03-12 03:23:11.343858026 +0000 UTC m=+0.392993206 container init a3c1de778e0477eda318c177b977e9ad97ff7a8fbb0dd2e799516c21d822124b (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public, maintainer=Paul Cuzner , release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., name=ubi8, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897) 2024-03-12T03:23:11.597 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 podman[37868]: 2024-03-12 03:23:11.349363509 +0000 UTC m=+0.398498688 container start a3c1de778e0477eda318c177b977e9ad97ff7a8fbb0dd2e799516c21d822124b (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a, build-date=2022-04-29T11:48:11.055897, io.openshift.expose-services=, description=Ceph Grafana Container, version=8.5, com.redhat.component=ubi8-container, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , release=239.1651231664, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, vendor=Red Hat, Inc.) 2024-03-12T03:23:11.597 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 bash[37868]: a3c1de778e0477eda318c177b977e9ad97ff7a8fbb0dd2e799516c21d822124b 2024-03-12T03:23:11.597 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 systemd[1]: Started Ceph grafana.a for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:23:11.597 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=settings t=2024-03-12T03:23:11.511182073Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-12T03:23:11.597 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=settings t=2024-03-12T03:23:11.511411975Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-12T03:23:11.597 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=settings t=2024-03-12T03:23:11.511439569Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-12T03:23:11.597 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=settings t=2024-03-12T03:23:11.511452426Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-12T03:23:11.597 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=settings t=2024-03-12T03:23:11.511463377Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-12T03:23:11.598 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=settings t=2024-03-12T03:23:11.51147504Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-12T03:23:11.598 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=settings t=2024-03-12T03:23:11.511485739Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-12T03:23:11.598 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=settings t=2024-03-12T03:23:11.511496294Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-12T03:23:11.598 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=settings t=2024-03-12T03:23:11.511510945Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-12T03:23:11.598 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=settings t=2024-03-12T03:23:11.511521937Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-12T03:23:11.599 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=settings t=2024-03-12T03:23:11.511533085Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-12T03:23:11.599 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=settings t=2024-03-12T03:23:11.511543916Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-12T03:23:11.599 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=settings t=2024-03-12T03:23:11.511554538Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-12T03:23:11.599 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=settings t=2024-03-12T03:23:11.511566022Z level=info msg="App mode production" 2024-03-12T03:23:11.599 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=sqlstore t=2024-03-12T03:23:11.511627475Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-12T03:23:11.599 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=sqlstore t=2024-03-12T03:23:11.511653863Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-12T03:23:11.599 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.512787337Z level=info msg="Starting DB migrations" 2024-03-12T03:23:11.599 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.512882188Z level=info msg="Executing migration" id="create migration_log table" 2024-03-12T03:23:11.599 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.513431337Z level=info msg="Executing migration" id="create user table" 2024-03-12T03:23:11.599 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.513900853Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-12T03:23:11.599 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.514367515Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-12T03:23:11.600 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.514843814Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-12T03:23:11.600 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.515418008Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-12T03:23:11.600 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.515812706Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-12T03:23:11.600 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.516514923Z level=info msg="Executing migration" id="create user table v2" 2024-03-12T03:23:11.600 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.516911309Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-12T03:23:11.600 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.517309868Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-12T03:23:11.600 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.517758371Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-12T03:23:11.600 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.518157555Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-12T03:23:11.600 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.518540388Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-12T03:23:11.600 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.518965323Z level=info msg="Executing migration" id="Update user table charset" 2024-03-12T03:23:11.600 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.519305943Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-12T03:23:11.600 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.51976237Z level=info msg="Executing migration" id="Add missing user data" 2024-03-12T03:23:11.600 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.520150839Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-12T03:23:11.600 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.520600462Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-12T03:23:11.601 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.521020643Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-12T03:23:11.601 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.52146687Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-12T03:23:11.601 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.522651018Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-12T03:23:11.601 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.523138033Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-12T03:23:11.601 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.523700802Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-12T03:23:11.601 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.524271029Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-12T03:23:11.601 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.525108423Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-12T03:23:11.601 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.525969034Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-12T03:23:11.601 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.526780932Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-12T03:23:11.601 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.527645654Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-12T03:23:11.601 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.528501742Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-12T03:23:11.601 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.529395457Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-12T03:23:11.601 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.530220633Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-12T03:23:11.601 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.53109388Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-12T03:23:11.602 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.532012347Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-12T03:23:11.602 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.532791159Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-12T03:23:11.602 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.533655558Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-12T03:23:11.602 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.534473835Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-12T03:23:11.602 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.537265135Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-12T03:23:11.602 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.54028078Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-12T03:23:11.602 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.540767392Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-12T03:23:11.602 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.543615497Z level=info msg="Executing migration" id="create star table" 2024-03-12T03:23:11.602 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.546279676Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-12T03:23:11.602 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.549001513Z level=info msg="Executing migration" id="create org table v1" 2024-03-12T03:23:11.602 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.551638408Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-12T03:23:11.602 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.554508637Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-12T03:23:11.602 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.557038563Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-12T03:23:11.603 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.557499381Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-12T03:23:11.603 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.560132232Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-12T03:23:11.603 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.560572752Z level=info msg="Executing migration" id="Update org table charset" 2024-03-12T03:23:11.603 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.563122105Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-12T03:23:11.603 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.563462708Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-12T03:23:11.603 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.563759558Z level=info msg="Executing migration" id="create dashboard table" 2024-03-12T03:23:11.603 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.564159553Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-12T03:23:11.603 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.56679208Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-12T03:23:11.603 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.569233184Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-12T03:23:11.603 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.572490201Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-12T03:23:11.603 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.572934827Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-12T03:23:11.603 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.57557438Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-12T03:23:11.603 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.578379778Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-12T03:23:11.603 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.580887571Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-12T03:23:11.604 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.581327194Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-12T03:23:11.604 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.583970464Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-12T03:23:11.604 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.586634135Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-12T03:23:11.604 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.589221078Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-12T03:23:11.604 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.589529263Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-12T03:23:11.604 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.592165938Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-12T03:23:11.604 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.592819358Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-12T03:23:11.604 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.594933363Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-12T03:23:11.604 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.597271788Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-12T03:23:11.847 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.597801294Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-12T03:23:11.847 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.600274526Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-12T03:23:11.847 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.603021221Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-12T03:23:11.847 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.605481943Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-12T03:23:11.847 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.605797194Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-12T03:23:11.847 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.606579502Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-12T03:23:11.847 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.607072809Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-12T03:23:11.847 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.607894167Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-12T03:23:11.847 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.60841189Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-12T03:23:11.848 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.609323334Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-12T03:23:11.848 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.610006905Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-12T03:23:11.848 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.610787311Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-12T03:23:11.848 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.611060629Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-12T03:23:11.848 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.611490014Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-12T03:23:11.848 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.611880993Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-12T03:23:11.848 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.612782242Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-12T03:23:11.848 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.613633572Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-12T03:23:11.848 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.614119015Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-12T03:23:11.848 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.614934558Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-12T03:23:11.848 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.615737095Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-12T03:23:11.848 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.616619702Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-12T03:23:11.848 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.617337978Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-12T03:23:11.848 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.618124241Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-12T03:23:11.849 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.619123448Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-12T03:23:11.849 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.619899886Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-12T03:23:11.849 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.62033336Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-12T03:23:11.849 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.621240813Z level=info msg="Executing migration" id="create data_source table" 2024-03-12T03:23:11.849 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.622254788Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-12T03:23:11.849 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.622828256Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-12T03:23:11.849 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.623293324Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-12T03:23:11.849 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.62411775Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-12T03:23:11.849 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.624937914Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-12T03:23:11.849 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.625969302Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-12T03:23:11.849 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.627124386Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-12T03:23:11.849 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.627700573Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-12T03:23:11.849 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.62851661Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-12T03:23:11.849 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.629520676Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-12T03:23:11.849 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.63033202Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-12T03:23:11.851 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.631061141Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-12T03:23:11.851 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.632027135Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-12T03:23:11.851 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.632667224Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-12T03:23:11.851 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.635465174Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-12T03:23:11.851 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.638309513Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-12T03:23:11.851 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.641315406Z level=info msg="Executing migration" id="Add uid column" 2024-03-12T03:23:11.851 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.644128187Z level=info msg="Executing migration" id="Update uid value" 2024-03-12T03:23:11.851 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.646771721Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-12T03:23:11.851 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.64955221Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-12T03:23:11.851 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.652275615Z level=info msg="Executing migration" id="create api_key table" 2024-03-12T03:23:11.851 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.654908654Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-12T03:23:11.852 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.657536585Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-12T03:23:11.852 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.66026198Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-12T03:23:11.852 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.663126711Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-12T03:23:11.852 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.665705003Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-12T03:23:11.852 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.666290314Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-12T03:23:11.852 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.669017169Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-12T03:23:11.852 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.671734839Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-12T03:23:11.852 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.672212836Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-12T03:23:11.852 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.674870842Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-12T03:23:11.852 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.677566167Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-12T03:23:11.852 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.680039729Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-12T03:23:11.852 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.680369619Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-12T03:23:11.852 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.682941796Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-12T03:23:11.852 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.68324565Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-12T03:23:11.853 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.685539964Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-12T03:23:11.853 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.686085745Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-12T03:23:11.853 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.688204781Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-12T03:23:11.853 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.688745973Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-12T03:23:11.853 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.691485256Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-12T03:23:11.853 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.693901467Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-12T03:23:11.853 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.696698271Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-12T03:23:11.853 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.697217481Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-12T03:23:11.853 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.699723841Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-12T03:23:11.853 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.702635529Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-12T03:23:11.853 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.705086512Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-12T03:23:11.853 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.705486377Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-12T03:23:11.853 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.708125424Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-12T03:23:11.853 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.708697129Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-12T03:23:11.853 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.709257618Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-12T03:23:11.854 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.710095157Z level=info msg="Executing migration" id="create quota table v1" 2024-03-12T03:23:11.854 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.710489555Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-12T03:23:11.854 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.711468721Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-12T03:23:11.854 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.712116178Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-12T03:23:11.854 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.713162022Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-12T03:23:11.854 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.713720121Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-12T03:23:11.854 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.714609184Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-12T03:23:11.854 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.715323017Z level=info msg="Executing migration" id="create session table" 2024-03-12T03:23:11.854 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.716124868Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-12T03:23:11.854 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.716422105Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-12T03:23:11.854 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.717309417Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-12T03:23:11.854 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.71786713Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-12T03:23:11.854 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.718248926Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-12T03:23:11.854 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.719191619Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-12T03:23:11.855 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.719669404Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-12T03:23:11.855 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.720564425Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-12T03:23:11.855 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.720900328Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-12T03:23:11.855 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.721329868Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-12T03:23:11.855 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.722169608Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-12T03:23:11.855 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.722849256Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-12T03:23:11.855 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.723695994Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-12T03:23:11.855 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.724945603Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-12T03:23:11.855 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.7258219Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-12T03:23:11.855 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.726152431Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-12T03:23:11.855 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.726947064Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-12T03:23:11.855 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.727814151Z level=info msg="Executing migration" id="create alert table v1" 2024-03-12T03:23:11.855 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.7285539Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-12T03:23:11.855 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.729063853Z level=info msg="Executing migration" id="add index alert state" 2024-03-12T03:23:11.856 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.729843026Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-12T03:23:11.856 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.730778665Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-12T03:23:11.856 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.731476105Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-12T03:23:11.856 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.731935723Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-12T03:23:11.856 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.732875117Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-12T03:23:11.856 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.735463177Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-12T03:23:11.856 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.738231877Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-12T03:23:11.856 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.738692562Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-12T03:23:11.856 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.741384833Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-12T03:23:11.856 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.741927044Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-12T03:23:11.856 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.744228551Z level=info msg="Executing migration" id="Add column is_default" 2024-03-12T03:23:11.856 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.747010576Z level=info msg="Executing migration" id="Add column frequency" 2024-03-12T03:23:11.856 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.747823244Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-12T03:23:11.857 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.750488302Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-12T03:23:11.857 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.753185286Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-12T03:23:11.857 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.755813261Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-12T03:23:11.857 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.756190361Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-12T03:23:11.857 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.759067865Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-12T03:23:11.857 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.761436057Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-12T03:23:11.857 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.764133187Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-12T03:23:11.857 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.764541573Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-12T03:23:11.857 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.767218394Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-12T03:23:11.857 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.767801917Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-12T03:23:11.857 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.770034945Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-12T03:23:11.857 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.772507674Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-12T03:23:11.857 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.772856642Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-12T03:23:11.857 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.775654235Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-12T03:23:11.857 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.777961326Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-12T03:23:11.858 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.780569797Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-12T03:23:11.858 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.780886853Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-12T03:23:11.858 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.783660365Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-12T03:23:11.858 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.786170499Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-12T03:23:11.858 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.788737422Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-12T03:23:11.858 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.789319112Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-12T03:23:11.858 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.792089842Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-12T03:23:11.858 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.794435649Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-12T03:23:11.858 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.797065596Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-12T03:23:11.858 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.797607788Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-12T03:23:11.858 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.800211104Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-12T03:23:11.858 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.802795745Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-12T03:23:11.858 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.803479601Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-12T03:23:11.858 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.803903474Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-12T03:23:11.859 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.804691516Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-12T03:23:11.859 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.805686789Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-12T03:23:11.859 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.806446325Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-12T03:23:11.859 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.806855794Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-12T03:23:11.859 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.808421324Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-12T03:23:11.859 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.809198174Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-12T03:23:11.859 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.810120269Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-12T03:23:11.859 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.810628575Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-12T03:23:11.859 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.81141173Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-12T03:23:11.859 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.811740355Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-12T03:23:11.859 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.812428711Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-12T03:23:11.859 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.813202609Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-12T03:23:11.859 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.813635247Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-12T03:23:11.859 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.814482966Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-12T03:23:11.860 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.815237245Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-12T03:23:11.860 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.816054376Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-12T03:23:11.860 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.816491869Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-12T03:23:11.860 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.817298571Z level=info msg="Executing migration" id="Move region to single row" 2024-03-12T03:23:11.860 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.817673979Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-12T03:23:11.860 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.818118486Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-12T03:23:11.860 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.818535714Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-12T03:23:11.860 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.81892987Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-12T03:23:11.860 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.819361875Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-12T03:23:11.860 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.819740732Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-12T03:23:11.860 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.820188355Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-12T03:23:11.860 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.820542835Z level=info msg="Executing migration" id="create test_data table" 2024-03-12T03:23:11.860 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.821363454Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-12T03:23:11.860 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.822173371Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-12T03:23:11.861 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.822618637Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-12T03:23:11.861 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.823420936Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-12T03:23:11.861 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.824203443Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-12T03:23:11.861 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.825163714Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-12T03:23:11.861 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.825519439Z level=info msg="Executing migration" id="create team table" 2024-03-12T03:23:11.861 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.828341943Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-12T03:23:11.861 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.831749119Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-12T03:23:11.861 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.833339645Z level=info msg="Executing migration" id="create team member table" 2024-03-12T03:23:11.861 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.833921923Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-12T03:23:11.861 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.836777376Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-12T03:23:11.861 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.839434717Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-12T03:23:11.861 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.842032962Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-12T03:23:11.861 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.842736193Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-12T03:23:11.861 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.845284516Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-12T03:23:11.861 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.848006128Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-12T03:23:12.098 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.850619824Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-12T03:23:12.098 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.851140229Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-12T03:23:12.098 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.854035876Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-12T03:23:12.099 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.854655256Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-12T03:23:12.099 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.857302927Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-12T03:23:12.099 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.857753299Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-12T03:23:12.099 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.858219005Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-12T03:23:12.099 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.85867192Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-12T03:23:12.099 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.859096169Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-12T03:23:12.099 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.859433732Z level=info msg="Executing migration" id="create tag table" 2024-03-12T03:23:12.099 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.85984109Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-12T03:23:12.099 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.860309128Z level=info msg="Executing migration" id="create login attempt table" 2024-03-12T03:23:12.099 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.860724067Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-12T03:23:12.099 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.86118338Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-12T03:23:12.099 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.861618046Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-12T03:23:12.099 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.863504936Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-12T03:23:12.099 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.863931728Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-12T03:23:12.100 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.864395694Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-12T03:23:12.100 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.864725537Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-12T03:23:12.100 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.865204736Z level=info msg="Executing migration" id="create user auth table" 2024-03-12T03:23:12.100 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.865678463Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-12T03:23:12.100 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.866242937Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-12T03:23:12.100 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.866642924Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-12T03:23:12.100 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.867525357Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-12T03:23:12.100 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.868346721Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-12T03:23:12.100 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.869160036Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-12T03:23:12.100 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.869938158Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-12T03:23:12.100 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.87041156Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-12T03:23:12.100 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.871193249Z level=info msg="Executing migration" id="create server_lock table" 2024-03-12T03:23:12.100 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.871628793Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-12T03:23:12.101 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.872114162Z level=info msg="Executing migration" id="create user auth token table" 2024-03-12T03:23:12.101 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.872578908Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-12T03:23:12.101 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.87517738Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-12T03:23:12.101 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.875711562Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-12T03:23:12.101 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.876183635Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-12T03:23:12.101 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.877001624Z level=info msg="Executing migration" id="create cache_data table" 2024-03-12T03:23:12.101 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.877445607Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-12T03:23:12.101 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.877980976Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-12T03:23:12.102 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.878439787Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-12T03:23:12.102 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.88106602Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-12T03:23:12.102 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.881538788Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-12T03:23:12.102 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.881933518Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-12T03:23:12.102 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.882588441Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-12T03:23:12.103 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.885167624Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-12T03:23:12.103 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.8856026Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-12T03:23:12.103 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.885882359Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-12T03:23:12.103 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.88631274Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-12T03:23:12.103 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.886734927Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-12T03:23:12.103 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.889428425Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-12T03:23:12.103 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.892026044Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-12T03:23:12.103 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.892964563Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-12T03:23:12.103 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.893441139Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-12T03:23:12.103 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.893829423Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-12T03:23:12.103 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.896122814Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-12T03:23:12.103 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.898660751Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-12T03:23:12.103 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.901188718Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-12T03:23:12.103 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.901499625Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-12T03:23:12.103 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.903618935Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-12T03:23:12.104 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.904331713Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-12T03:23:12.104 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.906371776Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-12T03:23:12.104 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.908922252Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-12T03:23:12.104 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.909754124Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-12T03:23:12.104 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.910191337Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-12T03:23:12.104 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.912949055Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-12T03:23:12.104 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.916391204Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-12T03:23:12.104 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.919769261Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-12T03:23:12.104 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.920221264Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-12T03:23:12.104 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.92285202Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-12T03:23:12.104 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.925136182Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-12T03:23:12.104 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.925571015Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-12T03:23:12.104 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.926444361Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-12T03:23:12.104 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.927221924Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-12T03:23:12.104 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.928155545Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-12T03:23:12.105 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.928802257Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-12T03:23:12.105 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.929801806Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-12T03:23:12.105 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.930800306Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-12T03:23:12.105 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.931969253Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-12T03:23:12.105 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.932782372Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-12T03:23:12.105 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.933625075Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-12T03:23:12.105 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.934469913Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-12T03:23:12.105 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.935296187Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-12T03:23:12.105 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.936091845Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-12T03:23:12.105 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.936931806Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-12T03:23:12.105 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.937775691Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-12T03:23:12.105 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.93871618Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-12T03:23:12.105 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.939302479Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-12T03:23:12.105 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.939776346Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-12T03:23:12.106 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.94054706Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-12T03:23:12.106 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.940854146Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-12T03:23:12.106 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.941762563Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-12T03:23:12.106 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.942649093Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-12T03:23:12.106 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.94351448Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-12T03:23:12.106 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.944402257Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-12T03:23:12.106 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.945247098Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-12T03:23:12.106 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.946284284Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-12T03:23:12.106 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.946928308Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-12T03:23:12.106 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.947787182Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-12T03:23:12.106 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.949622227Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-12T03:23:12.106 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.950586325Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-12T03:23:12.106 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.951033954Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-12T03:23:12.107 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.951882883Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-12T03:23:12.107 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.952912698Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-12T03:23:12.107 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.953628035Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-12T03:23:12.107 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.954506382Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-12T03:23:12.107 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.955364157Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-12T03:23:12.107 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.956138901Z level=info msg="Executing migration" id="create alert_image table" 2024-03-12T03:23:12.107 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.958937854Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-12T03:23:12.107 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.961879674Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-12T03:23:12.107 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.964699565Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-12T03:23:12.107 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.967468352Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-12T03:23:12.107 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.967899161Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-12T03:23:12.107 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.967981205Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-12T03:23:12.107 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.970659401Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-12T03:23:12.108 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.973553176Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-12T03:23:12.108 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.976148759Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-12T03:23:12.108 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.976274299Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-12T03:23:12.108 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.976405354Z level=info msg="alerts found to migrate" alerts=0 2024-03-12T03:23:12.108 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.979070428Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-12T03:23:12.108 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.9816957Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-12T03:23:12.108 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.984335808Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-12T03:23:12.108 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.987247168Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-12T03:23:12.108 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.989761654Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-12T03:23:12.108 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.992522205Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-12T03:23:12.108 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.994996239Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-12T03:23:12.108 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:11 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:11.997688005Z level=info msg="Executing migration" id="create data_keys table" 2024-03-12T03:23:12.108 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.000271037Z level=info msg="Executing migration" id="create secrets table" 2024-03-12T03:23:12.108 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.002992267Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-12T03:23:12.109 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.007573932Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-12T03:23:12.109 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.010281283Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-12T03:23:12.109 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.013051776Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-12T03:23:12.109 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.017412516Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-12T03:23:12.109 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.021725799Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-12T03:23:12.109 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.024351133Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-12T03:23:12.109 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.024839769Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-12T03:23:12.109 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.025231193Z level=info msg="Executing migration" id="create permission table" 2024-03-12T03:23:12.109 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.02566253Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-12T03:23:12.109 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.02613729Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-12T03:23:12.109 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.026785943Z level=info msg="Executing migration" id="create role table" 2024-03-12T03:23:12.109 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.027230583Z level=info msg="Executing migration" id="add column display_name" 2024-03-12T03:23:12.109 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.029997256Z level=info msg="Executing migration" id="add column group_name" 2024-03-12T03:23:12.109 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.032611379Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-12T03:23:12.110 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.035116809Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-12T03:23:12.110 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.037743359Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-12T03:23:12.110 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.040212151Z level=info msg="Executing migration" id="create team role table" 2024-03-12T03:23:12.110 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.042872231Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-12T03:23:12.110 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.045528771Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-12T03:23:12.110 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.048072271Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-12T03:23:12.110 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.050726822Z level=info msg="Executing migration" id="create user role table" 2024-03-12T03:23:12.110 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.053249599Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-12T03:23:12.110 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.055889216Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-12T03:23:12.110 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.058287919Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-12T03:23:12.110 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.060825951Z level=info msg="Executing migration" id="create builtin role table" 2024-03-12T03:23:12.110 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.063206016Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-12T03:23:12.110 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.065729417Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-12T03:23:12.111 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.068304226Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-12T03:23:12.111 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.071011231Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-12T03:23:12.111 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.0736163Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-12T03:23:12.111 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.076180872Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-12T03:23:12.111 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.078854977Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-12T03:23:12.111 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.081292301Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-12T03:23:12.111 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.083927245Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-12T03:23:12.111 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.086447744Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-12T03:23:12.111 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.088893742Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-12T03:23:12.111 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.091302191Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-12T03:23:12.111 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.093879531Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-12T03:23:12.111 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.096375539Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-12T03:23:12.111 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.09899928Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-12T03:23:12.271 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:12 smithi007 ceph-mon[25800]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:12.271 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:12 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:12.271 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:12 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:12.271 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:12 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:12.271 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:12 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:12.352 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-mon[27552]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:12.352 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:12.352 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:12.352 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:12.352 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:12.352 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.10147712Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-12T03:23:12.352 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.102238294Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-12T03:23:12.352 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.103022925Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-12T03:23:12.353 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.103491283Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-12T03:23:12.353 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.104344664Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-12T03:23:12.353 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.105101341Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-12T03:23:12.353 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.106102791Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-12T03:23:12.353 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.106885805Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-12T03:23:12.353 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.107711831Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-12T03:23:12.353 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.10848305Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-12T03:23:12.353 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.10927578Z level=info msg="Executing migration" id="add correlation config column" 2024-03-12T03:23:12.353 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.110269757Z level=info msg="Executing migration" id="create entity_events table" 2024-03-12T03:23:12.353 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.111100482Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-12T03:23:12.353 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.111931151Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-12T03:23:12.353 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.112022019Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-12T03:23:12.353 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.112736226Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-12T03:23:12.353 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.112815119Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-12T03:23:12.354 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.113481833Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-12T03:23:12.354 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.113926009Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-12T03:23:12.354 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.114390177Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-12T03:23:12.354 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.11514597Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-12T03:23:12.354 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.115934997Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-12T03:23:12.354 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.11676993Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-12T03:23:12.354 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.117589429Z level=info msg="Executing migration" id="Drop public config table" 2024-03-12T03:23:12.354 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.118429621Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-12T03:23:12.354 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.11920267Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-12T03:23:12.354 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.120021271Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-12T03:23:12.354 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.120792377Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-12T03:23:12.354 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.121593954Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-12T03:23:12.354 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.12463705Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-12T03:23:12.354 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.125650041Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-12T03:23:12.355 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.126682861Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-12T03:23:12.355 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.12751549Z level=info msg="Executing migration" id="add share column" 2024-03-12T03:23:12.355 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.128530934Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-12T03:23:12.355 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.12934558Z level=info msg="Executing migration" id="create file table" 2024-03-12T03:23:12.355 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.13012651Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-12T03:23:12.355 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.130948852Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-12T03:23:12.355 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.131781005Z level=info msg="Executing migration" id="create file_meta table" 2024-03-12T03:23:12.356 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.134300888Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-12T03:23:12.356 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.137000633Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-12T03:23:12.356 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.139564703Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-12T03:23:12.356 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.14215868Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-12T03:23:12.356 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.14484833Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-12T03:23:12.356 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.14748922Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-12T03:23:12.356 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.150226819Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-12T03:23:12.356 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.152898271Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-12T03:23:12.356 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.155524895Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-12T03:23:12.356 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.158070122Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-12T03:23:12.356 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.160665195Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-12T03:23:12.356 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.16336701Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-12T03:23:12.356 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.165978959Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-12T03:23:12.356 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.168538085Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-12T03:23:12.357 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.171093359Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-12T03:23:12.357 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.181137503Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-12T03:23:12.357 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.183713095Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-12T03:23:12.357 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.186219852Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-12T03:23:12.357 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.189807678Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-12T03:23:12.357 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.192448644Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-12T03:23:12.357 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.195114646Z level=info msg="Executing migration" id="create folder table" 2024-03-12T03:23:12.357 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.197785456Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-12T03:23:12.357 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.200382265Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-12T03:23:12.357 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.203035245Z level=info msg="Executing migration" id="Update folder title length" 2024-03-12T03:23:12.357 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.205451461Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-12T03:23:12.357 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=migrator t=2024-03-12T03:23:12.208054335Z level=info msg="migrations completed" performed=485 skipped=0 duration=695.205869ms 2024-03-12T03:23:12.357 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=sqlstore t=2024-03-12T03:23:12.2084787Z level=info msg="Created default organization" 2024-03-12T03:23:12.357 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=plugin.loader t=2024-03-12T03:23:12.271193349Z level=info msg="Plugin registered" pluginID=input 2024-03-12T03:23:12.358 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=plugin.loader t=2024-03-12T03:23:12.303618234Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-12T03:23:12.358 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=plugin.loader t=2024-03-12T03:23:12.303647576Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-12T03:23:12.358 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=secrets t=2024-03-12T03:23:12.303689725Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-12T03:23:12.358 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=query_data t=2024-03-12T03:23:12.306195036Z level=info msg="Query Service initialization" 2024-03-12T03:23:12.358 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:12 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=live.push_http t=2024-03-12T03:23:12.308662515Z level=info msg="Live Push Gateway initialization" 2024-03-12T03:23:12.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:12 smithi170 ceph-mon[27708]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:12.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:12 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:12.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:12 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:12.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:12 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:12.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:12 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:12.393 INFO:teuthology.orchestra.run.smithi007.stdout:Backend: cephadm 2024-03-12T03:23:12.394 INFO:teuthology.orchestra.run.smithi007.stdout:Available: Yes 2024-03-12T03:23:12.394 INFO:teuthology.orchestra.run.smithi007.stdout:Paused: No 2024-03-12T03:23:13.148 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-12T03:23:13.346 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-mon[27552]: Deploying daemon node-exporter.a on smithi007 2024-03-12T03:23:13.347 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=infra.usagestats.collector t=2024-03-12T03:23:13.043812707Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-12T03:23:13.347 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=provisioning.datasources t=2024-03-12T03:23:13.044777241Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-12T03:23:13.347 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=provisioning.datasources t=2024-03-12T03:23:13.056859996Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-12T03:23:13.347 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=provisioning.alerting t=2024-03-12T03:23:13.068173704Z level=info msg="starting to provision alerting" 2024-03-12T03:23:13.347 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=provisioning.alerting t=2024-03-12T03:23:13.06819342Z level=info msg="finished to provision alerting" 2024-03-12T03:23:13.347 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=ngalert.state.manager t=2024-03-12T03:23:13.068259324Z level=info msg="Warming state cache for startup" 2024-03-12T03:23:13.347 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=grafanaStorageLogger t=2024-03-12T03:23:13.068333804Z level=info msg="storage starting" 2024-03-12T03:23:13.348 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=http.server t=2024-03-12T03:23:13.070780405Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-12T03:23:13.348 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=ngalert.state.manager t=2024-03-12T03:23:13.072760907Z level=info msg="State cache has been initialized" states=0 duration=4.501704ms 2024-03-12T03:23:13.348 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=ngalert.multiorg.alertmanager t=2024-03-12T03:23:13.072811524Z level=info msg="starting MultiOrg Alertmanager" 2024-03-12T03:23:13.348 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=ticker t=2024-03-12T03:23:13.072837297Z level=info msg=starting first_tick=2024-03-12T03:23:20Z 2024-03-12T03:23:13.348 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=sqlstore.transactions t=2024-03-12T03:23:13.082219944Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-12T03:23:13.348 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=sqlstore.transactions t=2024-03-12T03:23:13.09907745Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-12T03:23:13.348 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=sqlstore.transactions t=2024-03-12T03:23:13.099089193Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-12T03:23:13.348 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=sqlstore.transactions t=2024-03-12T03:23:13.109350785Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-12T03:23:13.348 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=sqlstore.transactions t=2024-03-12T03:23:13.119842269Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-03-12T03:23:13.348 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:23:13 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a[37883]: logger=sqlstore.transactions t=2024-03-12T03:23:13.130361326Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4 code="database is locked" 2024-03-12T03:23:13.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:13 smithi170 ceph-mon[27708]: Deploying daemon node-exporter.a on smithi007 2024-03-12T03:23:13.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:13 smithi007 ceph-mon[25800]: Deploying daemon node-exporter.a on smithi007 2024-03-12T03:23:13.407 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:23:14.277 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:14 smithi007 ceph-mon[25800]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:14.278 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:14 smithi007 ceph-mon[25800]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:14.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:14 smithi170 ceph-mon[27708]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:14.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:14 smithi170 ceph-mon[27708]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:14.377 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:14 smithi033 ceph-mon[27552]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:14.377 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:14 smithi033 ceph-mon[27552]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:15.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:15 smithi007 ceph-mon[25800]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:15.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:15 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:15.677 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:15 smithi033 ceph-mon[27552]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:15.678 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:15 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:15.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:15 smithi170 ceph-mon[27708]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:15.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:15 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:15.948 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:15 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: ::ffff:172.21.15.33 - - [12/Mar/2024:03:23:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:23:16.670 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:23:16.670 INFO:teuthology.orchestra.run.smithi007.stdout:agent.smithi007 smithi007 running 49s ago 3m - - 2024-03-12T03:23:16.670 INFO:teuthology.orchestra.run.smithi007.stdout:agent.smithi033 smithi033 running 61s ago 3m - - 2024-03-12T03:23:16.670 INFO:teuthology.orchestra.run.smithi007.stdout:agent.smithi170 smithi170 running 62s ago 3m - - 2024-03-12T03:23:16.670 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 starting - - - - 2024-03-12T03:23:16.670 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi033 *:3000 starting - - - - 2024-03-12T03:23:16.671 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi033 *:8443,8765 running (2m) 61s ago 2m 450M - 19.0.0-1979-g2b0101eb 4133f8bc41bb ec8321b1b7ab 2024-03-12T03:23:16.671 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:9283,8765 running (4m) 49s ago 4m 514M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 42b9d300140d 2024-03-12T03:23:16.671 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.z smithi170 *:8443,8765 running (2m) 62s ago 2m 451M - 19.0.0-1979-g2b0101eb 4133f8bc41bb fe9479a3eaa9 2024-03-12T03:23:16.671 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (4m) 49s ago 4m 46.4M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb d8b3346c0808 2024-03-12T03:23:16.671 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi033 running (3m) 61s ago 3m 40.0M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 8147033ac082 2024-03-12T03:23:16.671 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi170 running (3m) 62s ago 3m 41.0M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb a8d1056bc966 2024-03-12T03:23:16.671 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (2m) 49s ago 2m 62.8M 4096M 19.0.0-1979-g2b0101eb 4133f8bc41bb 74794dbb8ec2 2024-03-12T03:23:16.671 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi033 running (2m) 61s ago 2m 61.3M 4096M 19.0.0-1979-g2b0101eb 4133f8bc41bb 98c4011e73f3 2024-03-12T03:23:16.671 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi170 running (95s) 62s ago 95s 60.2M 4096M 19.0.0-1979-g2b0101eb 4133f8bc41bb 07edd3bf199d 2024-03-12T03:23:16.671 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi033 *:9095 starting - - - - 2024-03-12T03:23:16.671 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.a smithi007 *:80 running (74s) 49s ago 73s 77.1M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 6018bc187853 2024-03-12T03:23:17.406 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:17 smithi170 ceph-mon[27708]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:17.407 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:17 smithi170 ceph-mon[27708]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:17.407 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:17 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:17.407 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:17 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:17.407 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:17 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:17.634 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:17 smithi007 ceph-mon[25800]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:17.634 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:17 smithi007 ceph-mon[25800]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:17.634 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:17 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:17.634 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:17 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:17.634 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:17 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:17.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:17 smithi033 ceph-mon[27552]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:17.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:17 smithi033 ceph-mon[27552]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:17.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:17 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:17.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:17 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:17.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:17 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:18.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-12T03:23:18.463 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:18 smithi033 systemd[1]: Starting Ceph node-exporter.b for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:23:18.560 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:23:18.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:18 smithi033 ceph-mon[27552]: Deploying daemon node-exporter.b on smithi033 2024-03-12T03:23:18.763 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:18 smithi033 bash[38899]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-12T03:23:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:18 smithi007 ceph-mon[25800]: Deploying daemon node-exporter.b on smithi033 2024-03-12T03:23:19.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:18 smithi170 ceph-mon[27708]: Deploying daemon node-exporter.b on smithi033 2024-03-12T03:23:19.263 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:18 smithi033 bash[38899]: Getting image source signatures 2024-03-12T03:23:19.263 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:18 smithi033 bash[38899]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-12T03:23:19.263 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:18 smithi033 bash[38899]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-12T03:23:19.263 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:18 smithi033 bash[38899]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-12T03:23:19.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:19 smithi007 ceph-mon[25800]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:19.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:19 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:20.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:19 smithi033 ceph-mon[27552]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:20.013 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:19 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:20.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:19 smithi170 ceph-mon[27708]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:20.113 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:19 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:20.513 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 bash[38899]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-12T03:23:20.513 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 bash[38899]: Writing manifest to image destination 2024-03-12T03:23:20.909 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 podman[38899]: 2024-03-12 03:23:20.498994976 +0000 UTC m=+2.132283274 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-12T03:23:20.909 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 podman[38899]: 2024-03-12 03:23:20.614395089 +0000 UTC m=+2.247683380 container create e23ad178c456c6fe3b3aa1faf6275bcf8c31eeb14b79a5f5c53443b0b8751034 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:23:20.910 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 podman[38899]: 2024-03-12 03:23:20.807929157 +0000 UTC m=+2.441217444 container init e23ad178c456c6fe3b3aa1faf6275bcf8c31eeb14b79a5f5c53443b0b8751034 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:23:20.910 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 podman[38899]: 2024-03-12 03:23:20.813426906 +0000 UTC m=+2.446715194 container start e23ad178c456c6fe3b3aa1faf6275bcf8c31eeb14b79a5f5c53443b0b8751034 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:23:20.910 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.817Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-12T03:23:20.910 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.817Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-12T03:23:20.910 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.817Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-12T03:23:20.910 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.817Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-12T03:23:20.910 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-12T03:23:20.910 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-12T03:23:20.910 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-12T03:23:20.910 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=arp 2024-03-12T03:23:20.910 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-12T03:23:20.910 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-12T03:23:20.910 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-12T03:23:20.911 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-12T03:23:20.911 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-12T03:23:20.911 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-12T03:23:20.911 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-12T03:23:20.911 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-12T03:23:20.911 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=edac 2024-03-12T03:23:20.911 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-12T03:23:20.911 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-12T03:23:20.912 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-12T03:23:20.912 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-12T03:23:20.912 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-12T03:23:20.912 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-12T03:23:20.912 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-12T03:23:20.912 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-12T03:23:20.912 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-12T03:23:20.912 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-12T03:23:20.912 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-12T03:23:20.912 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-12T03:23:20.912 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-12T03:23:20.912 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-12T03:23:20.912 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-12T03:23:20.912 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-12T03:23:20.913 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=os 2024-03-12T03:23:20.913 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-12T03:23:20.913 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-12T03:23:20.913 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-12T03:23:20.913 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-12T03:23:20.913 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-12T03:23:20.913 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-12T03:23:20.913 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-12T03:23:20.913 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=stat 2024-03-12T03:23:20.913 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-12T03:23:20.913 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-12T03:23:20.913 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-12T03:23:20.913 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=time 2024-03-12T03:23:20.913 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-12T03:23:20.913 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=uname 2024-03-12T03:23:20.914 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-12T03:23:20.914 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-12T03:23:20.914 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.818Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-12T03:23:20.914 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.819Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-12T03:23:20.914 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b[38948]: ts=2024-03-12T03:23:20.819Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-12T03:23:20.914 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 bash[38899]: e23ad178c456c6fe3b3aa1faf6275bcf8c31eeb14b79a5f5c53443b0b8751034 2024-03-12T03:23:21.262 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:23:20 smithi033 systemd[1]: Started Ceph node-exporter.b for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:23:21.586 INFO:teuthology.orchestra.run.smithi007.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-12T03:23:21.587 INFO:teuthology.orchestra.run.smithi007.stdout:agent 3/3 67s ago 47s * 2024-03-12T03:23:21.587 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager ?:9093,9094 0/1 - 72s smithi007=a;count:1 2024-03-12T03:23:21.587 INFO:teuthology.orchestra.run.smithi007.stdout:grafana ?:3000 0/1 - 70s smithi033=a;count:1 2024-03-12T03:23:21.587 INFO:teuthology.orchestra.run.smithi007.stdout:mgr 3/3 67s ago 2m smithi007=y;smithi033=x;smithi170=z;count:3 2024-03-12T03:23:21.587 INFO:teuthology.orchestra.run.smithi007.stdout:mon 3/3 67s ago 3m smithi007:172.21.15.7=a;smithi033:172.21.15.33=b;smithi170:172.21.15.170=c;count:3 2024-03-12T03:23:21.587 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter ?:9100 0/3 - 75s smithi007=a;smithi033=b;smithi170=c;count:3 2024-03-12T03:23:21.587 INFO:teuthology.orchestra.run.smithi007.stdout:osd 3 67s ago - 2024-03-12T03:23:21.587 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus ?:9095 0/1 - 78s smithi033=a;count:1 2024-03-12T03:23:21.587 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.a ?:80 1/1 54s ago 78s smithi007=foo.a;count:1 2024-03-12T03:23:22.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:21 smithi033 ceph-mon[27552]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:22.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:21 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:22.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:21 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:22.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:21 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:22.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:21 smithi033 ceph-mon[27552]: Deploying daemon node-exporter.c on smithi170 2024-03-12T03:23:22.339 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:21 smithi007 ceph-mon[25800]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:22.339 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:21 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:22.339 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:21 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:22.339 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:21 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:22.339 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:21 smithi007 ceph-mon[25800]: Deploying daemon node-exporter.c on smithi170 2024-03-12T03:23:22.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:21 smithi170 ceph-mon[27708]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:22.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:21 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:22.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:21 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:22.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:21 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:22.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:21 smithi170 ceph-mon[27708]: Deploying daemon node-exporter.c on smithi170 2024-03-12T03:23:22.364 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:22 smithi170 bash[39252]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-12T03:23:22.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-12T03:23:22.863 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:22 smithi170 bash[39252]: Getting image source signatures 2024-03-12T03:23:22.863 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:22 smithi170 bash[39252]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-12T03:23:22.863 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:22 smithi170 bash[39252]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-12T03:23:22.864 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:22 smithi170 bash[39252]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-12T03:23:22.944 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:23:23.216 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:22 smithi007 ceph-mon[25800]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:23.262 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:22 smithi033 ceph-mon[27552]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:23.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:22 smithi170 ceph-mon[27708]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:23.965 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:23 smithi170 bash[39252]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-12T03:23:23.965 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:23 smithi170 bash[39252]: Writing manifest to image destination 2024-03-12T03:23:24.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:23 smithi007 ceph-mon[25800]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:24.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:23 smithi033 ceph-mon[27552]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:24.299 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:23 smithi170 ceph-mon[27708]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:24.299 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 podman[39252]: 2024-03-12 03:23:24.10843922 +0000 UTC m=+2.081886006 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-12T03:23:24.300 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 podman[39252]: 2024-03-12 03:23:24.206480853 +0000 UTC m=+2.179927632 container create d913010cc4adffe2ea75ef4b528570a8800bb85c8e8bcd966f1eb72e7c9d58c7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-12T03:23:24.864 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 podman[39252]: 2024-03-12 03:23:24.400127878 +0000 UTC m=+2.373574673 container init d913010cc4adffe2ea75ef4b528570a8800bb85c8e8bcd966f1eb72e7c9d58c7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-12T03:23:24.865 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 podman[39252]: 2024-03-12 03:23:24.405739642 +0000 UTC m=+2.379186421 container start d913010cc4adffe2ea75ef4b528570a8800bb85c8e8bcd966f1eb72e7c9d58c7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-12T03:23:24.865 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.410Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-12T03:23:24.865 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.410Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-12T03:23:24.865 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.410Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-12T03:23:24.865 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.410Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-12T03:23:24.865 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.410Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-12T03:23:24.866 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.410Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-12T03:23:24.866 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-12T03:23:24.866 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=arp 2024-03-12T03:23:24.866 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-12T03:23:24.866 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-12T03:23:24.866 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-12T03:23:24.867 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-12T03:23:24.867 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-12T03:23:24.867 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-12T03:23:24.867 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-12T03:23:24.868 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-12T03:23:24.868 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=edac 2024-03-12T03:23:24.868 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-12T03:23:24.868 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-12T03:23:24.868 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-12T03:23:24.868 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-12T03:23:24.868 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-12T03:23:24.869 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-12T03:23:24.869 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-12T03:23:24.869 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-12T03:23:24.869 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-12T03:23:24.869 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-12T03:23:24.869 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-12T03:23:24.870 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-12T03:23:24.870 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-12T03:23:24.870 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-12T03:23:24.870 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-12T03:23:24.870 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-12T03:23:24.870 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=os 2024-03-12T03:23:24.871 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-12T03:23:24.871 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-12T03:23:24.871 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-12T03:23:24.871 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-12T03:23:24.871 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-12T03:23:24.871 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-12T03:23:24.872 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-12T03:23:24.872 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=stat 2024-03-12T03:23:24.872 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-12T03:23:24.872 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-12T03:23:24.872 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-12T03:23:24.872 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=time 2024-03-12T03:23:24.872 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-12T03:23:24.872 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=uname 2024-03-12T03:23:24.873 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-12T03:23:24.873 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-12T03:23:24.873 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-12T03:23:24.873 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-12T03:23:24.873 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c[39302]: ts=2024-03-12T03:23:24.411Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-12T03:23:24.873 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 bash[39252]: d913010cc4adffe2ea75ef4b528570a8800bb85c8e8bcd966f1eb72e7c9d58c7 2024-03-12T03:23:24.873 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:23:24 smithi170 systemd[1]: Started Ceph node-exporter.c for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:23:25.017 INFO:teuthology.orchestra.run.smithi007.stdout:HOST ADDR LABELS STATUS 2024-03-12T03:23:25.018 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 172.21.15.7 2024-03-12T03:23:25.018 INFO:teuthology.orchestra.run.smithi007.stdout:smithi033 172.21.15.33 2024-03-12T03:23:25.018 INFO:teuthology.orchestra.run.smithi007.stdout:smithi170 172.21.15.170 2024-03-12T03:23:25.018 INFO:teuthology.orchestra.run.smithi007.stdout:3 hosts in cluster 2024-03-12T03:23:25.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:25 smithi007 ceph-mon[25800]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:25.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:25 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:25.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:25 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:25.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:25 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:25.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:25 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:25.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:25 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:25.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:25 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:23:25.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:25 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi033", "caps": []}]: dispatch 2024-03-12T03:23:25.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:25 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]: dispatch 2024-03-12T03:23:25.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:25 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi007", "caps": []}]: dispatch 2024-03-12T03:23:25.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:25 smithi007 ceph-mon[25800]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:25.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:25 smithi170 ceph-mon[27708]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:25.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:25 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:25.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:25 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:25.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:25 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:25.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:25 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:25.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:25 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:25.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:25 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:23:25.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:25 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi033", "caps": []}]: dispatch 2024-03-12T03:23:25.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:25 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]: dispatch 2024-03-12T03:23:25.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:25 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi007", "caps": []}]: dispatch 2024-03-12T03:23:25.864 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:25 smithi170 ceph-mon[27708]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:25.926 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:25 smithi033 ceph-mon[27552]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:25.926 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:25 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:25.926 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:25 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:25.926 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:25 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:25.926 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:25 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:25.926 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:25 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:25.926 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:25 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:23:25.926 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:25 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi033", "caps": []}]: dispatch 2024-03-12T03:23:25.926 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:25 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]: dispatch 2024-03-12T03:23:25.926 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:25 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi007", "caps": []}]: dispatch 2024-03-12T03:23:25.926 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:25 smithi033 ceph-mon[27552]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:26.075 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:25 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: ::ffff:172.21.15.33 - - [12/Mar/2024:03:23:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:23:26.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-12T03:23:27.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:27 smithi033 ceph-mon[27552]: Reconfiguring daemon agent.smithi170 on smithi170 2024-03-12T03:23:27.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:27 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:27.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:27 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:27.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:27 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:27.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:27 smithi007 ceph-mon[25800]: Reconfiguring daemon agent.smithi170 on smithi170 2024-03-12T03:23:27.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:27 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:27.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:27 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:27.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:27 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:27.600 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:27 smithi170 ceph-mon[27708]: Reconfiguring daemon agent.smithi170 on smithi170 2024-03-12T03:23:27.600 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:27 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:27.600 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:27 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:27.600 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:27 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:27.650 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:23:28.255 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:28 smithi170 ceph-mon[27708]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:28.255 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:28 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:28.255 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:28 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:28.255 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:28 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:28.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:28 smithi033 ceph-mon[27552]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:28.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:28 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:28.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:28 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:28.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:28 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:28.270 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:28 smithi007 ceph-mon[25800]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:28.271 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:28 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:28.271 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:28 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:28.271 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:28 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:29.179 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:29 smithi170 ceph-mon[27708]: Reconfiguring daemon agent.smithi033 on smithi033 2024-03-12T03:23:29.179 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:29 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:29.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:29 smithi007 ceph-mon[25800]: Reconfiguring daemon agent.smithi033 on smithi033 2024-03-12T03:23:29.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:29 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:29.410 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:29 smithi033 ceph-mon[27552]: Reconfiguring daemon agent.smithi033 on smithi033 2024-03-12T03:23:29.410 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:29 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:30.209 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:30 smithi170 ceph-mon[27708]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:30.210 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:30 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:30.210 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:30 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:30.210 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:30 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:30.210 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:30 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:30.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:30 smithi007 ceph-mon[25800]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:30.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:30 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:30.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:30 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:30.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:30 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:30.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:30 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:30.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:30 smithi033 ceph-mon[27552]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:30.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:30 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:30.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:30 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:30.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:30 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:30.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:30 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:31.087 INFO:teuthology.orchestra.run.smithi007.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-12T03:23:31.088 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300GE480DGN 447G No 83s ago Has a FileSystem, LVM detected 2024-03-12T03:23:31.088 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/nvme1n1 ssd Linux_cdb9c672fe801c2263e2 89.4G Yes 83s ago 2024-03-12T03:23:31.088 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/nvme2n1 ssd Linux_4b33db6b75080daa5b11 89.4G Yes 83s ago 2024-03-12T03:23:31.089 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/nvme3n1 ssd Linux_72c4ca82d801ee31dd06 89.4G Yes 83s ago 2024-03-12T03:23:31.089 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/nvme4n1 ssd Linux_e0d0d19c92fa36667d17 89.4G No 83s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-12T03:23:31.089 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W4HPND 931G No 83s ago Has partitions 2024-03-12T03:23:31.089 INFO:teuthology.orchestra.run.smithi007.stdout:smithi033 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330008Q400BGN 372G No 76s ago Has a FileSystem, LVM detected 2024-03-12T03:23:31.089 INFO:teuthology.orchestra.run.smithi007.stdout:smithi033 /dev/nvme1n1 ssd Linux_386594f789e6b989396c 89.4G Yes 76s ago 2024-03-12T03:23:31.089 INFO:teuthology.orchestra.run.smithi007.stdout:smithi033 /dev/nvme2n1 ssd Linux_acaba4db481862bf6cc5 89.4G Yes 76s ago 2024-03-12T03:23:31.089 INFO:teuthology.orchestra.run.smithi007.stdout:smithi033 /dev/nvme3n1 ssd Linux_c1863f6c2613c5dc9a48 89.4G Yes 76s ago 2024-03-12T03:23:31.089 INFO:teuthology.orchestra.run.smithi007.stdout:smithi033 /dev/nvme4n1 ssd Linux_a587843b9b7207625e5c 89.4G No 76s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-12T03:23:31.090 INFO:teuthology.orchestra.run.smithi007.stdout:smithi033 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W4HQ2L 931G No 76s ago Has partitions 2024-03-12T03:23:31.090 INFO:teuthology.orchestra.run.smithi007.stdout:smithi170 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476002R480DGN 447G No 77s ago Has a FileSystem, LVM detected 2024-03-12T03:23:31.090 INFO:teuthology.orchestra.run.smithi007.stdout:smithi170 /dev/nvme1n1 ssd Linux_345ab9225a67e14d3da0 89.4G Yes 77s ago 2024-03-12T03:23:31.090 INFO:teuthology.orchestra.run.smithi007.stdout:smithi170 /dev/nvme2n1 ssd Linux_5351117ee63a7d3743c0 89.4G Yes 77s ago 2024-03-12T03:23:31.090 INFO:teuthology.orchestra.run.smithi007.stdout:smithi170 /dev/nvme3n1 ssd Linux_1824f99b508c04c684b9 89.4G Yes 77s ago 2024-03-12T03:23:31.090 INFO:teuthology.orchestra.run.smithi007.stdout:smithi170 /dev/nvme4n1 ssd Linux_c9338f697901a0741dd1 89.4G No 77s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-12T03:23:31.090 INFO:teuthology.orchestra.run.smithi007.stdout:smithi170 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XK1Y 931G No 77s ago Has partitions 2024-03-12T03:23:31.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:31 smithi170 ceph-mon[27708]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:31.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:31 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:31.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:31 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:31.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:31 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:31.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:31 smithi170 ceph-mon[27708]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:31.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:31 smithi007 ceph-mon[25800]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:31.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:31 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:31.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:31 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:31.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:31 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:31.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:31 smithi007 ceph-mon[25800]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:31.662 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:31 smithi033 ceph-mon[27552]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:31.662 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:31 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:31.662 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:31 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:31.662 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:31 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:31.662 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:31 smithi033 ceph-mon[27552]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:32.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-12T03:23:32.846 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:23:33.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:33 smithi033 ceph-mon[27552]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:33.593 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:33 smithi170 ceph-mon[27708]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:33.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:33 smithi007 ceph-mon[25800]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:34.816 INFO:teuthology.orchestra.run.smithi007.stdout:service_type: agent 2024-03-12T03:23:34.816 INFO:teuthology.orchestra.run.smithi007.stdout:service_name: agent 2024-03-12T03:23:34.816 INFO:teuthology.orchestra.run.smithi007.stdout:placement: 2024-03-12T03:23:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: host_pattern: '*' 2024-03-12T03:23:34.816 INFO:teuthology.orchestra.run.smithi007.stdout:status: 2024-03-12T03:23:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: created: '2024-03-12T03:22:34.337911Z' 2024-03-12T03:23:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: last_refresh: '2024-03-12T03:23:33.996670Z' 2024-03-12T03:23:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: running: 2 2024-03-12T03:23:34.816 INFO:teuthology.orchestra.run.smithi007.stdout: size: 3 2024-03-12T03:23:34.816 INFO:teuthology.orchestra.run.smithi007.stdout:events: 2024-03-12T03:23:34.816 INFO:teuthology.orchestra.run.smithi007.stdout:- 2024-03-12T03:22:35.739761Z service:agent [INFO] "service was created" 2024-03-12T03:23:34.816 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2024-03-12T03:23:34.817 INFO:teuthology.orchestra.run.smithi007.stdout:service_type: alertmanager 2024-03-12T03:23:34.817 INFO:teuthology.orchestra.run.smithi007.stdout:service_name: alertmanager 2024-03-12T03:23:34.817 INFO:teuthology.orchestra.run.smithi007.stdout:placement: 2024-03-12T03:23:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: count: 1 2024-03-12T03:23:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: hosts: 2024-03-12T03:23:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: - smithi007=a 2024-03-12T03:23:34.817 INFO:teuthology.orchestra.run.smithi007.stdout:status: 2024-03-12T03:23:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: created: '2024-03-12T03:22:08.617021Z' 2024-03-12T03:23:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: last_refresh: '2024-03-12T03:23:30.301693Z' 2024-03-12T03:23:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: ports: 2024-03-12T03:23:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: - 9093 2024-03-12T03:23:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: - 9094 2024-03-12T03:23:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: running: 1 2024-03-12T03:23:34.817 INFO:teuthology.orchestra.run.smithi007.stdout: size: 1 2024-03-12T03:23:34.818 INFO:teuthology.orchestra.run.smithi007.stdout:events: 2024-03-12T03:23:34.818 INFO:teuthology.orchestra.run.smithi007.stdout:- 2024-03-12T03:22:55.909896Z service:alertmanager [INFO] "service was created" 2024-03-12T03:23:34.818 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2024-03-12T03:23:34.818 INFO:teuthology.orchestra.run.smithi007.stdout:service_type: grafana 2024-03-12T03:23:34.818 INFO:teuthology.orchestra.run.smithi007.stdout:service_name: grafana 2024-03-12T03:23:34.818 INFO:teuthology.orchestra.run.smithi007.stdout:placement: 2024-03-12T03:23:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: count: 1 2024-03-12T03:23:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: hosts: 2024-03-12T03:23:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: - smithi033=a 2024-03-12T03:23:34.818 INFO:teuthology.orchestra.run.smithi007.stdout:spec: 2024-03-12T03:23:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: anonymous_access: true 2024-03-12T03:23:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: protocol: https 2024-03-12T03:23:34.818 INFO:teuthology.orchestra.run.smithi007.stdout:status: 2024-03-12T03:23:34.818 INFO:teuthology.orchestra.run.smithi007.stdout: created: '2024-03-12T03:22:11.531952Z' 2024-03-12T03:23:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: ports: 2024-03-12T03:23:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: - 3000 2024-03-12T03:23:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: running: 0 2024-03-12T03:23:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: size: 1 2024-03-12T03:23:34.819 INFO:teuthology.orchestra.run.smithi007.stdout:events: 2024-03-12T03:23:34.819 INFO:teuthology.orchestra.run.smithi007.stdout:- 2024-03-12T03:23:11.492124Z service:grafana [INFO] "service was created" 2024-03-12T03:23:34.819 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2024-03-12T03:23:34.819 INFO:teuthology.orchestra.run.smithi007.stdout:service_type: mgr 2024-03-12T03:23:34.819 INFO:teuthology.orchestra.run.smithi007.stdout:service_name: mgr 2024-03-12T03:23:34.819 INFO:teuthology.orchestra.run.smithi007.stdout:placement: 2024-03-12T03:23:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: count: 3 2024-03-12T03:23:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: hosts: 2024-03-12T03:23:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: - smithi007=y 2024-03-12T03:23:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: - smithi033=x 2024-03-12T03:23:34.820 INFO:teuthology.orchestra.run.smithi007.stdout: - smithi170=z 2024-03-12T03:23:34.820 INFO:teuthology.orchestra.run.smithi007.stdout:status: 2024-03-12T03:23:34.820 INFO:teuthology.orchestra.run.smithi007.stdout: created: '2024-03-12T03:20:22.731381Z' 2024-03-12T03:23:34.820 INFO:teuthology.orchestra.run.smithi007.stdout: last_refresh: '2024-03-12T03:22:14.734487Z' 2024-03-12T03:23:34.820 INFO:teuthology.orchestra.run.smithi007.stdout: running: 3 2024-03-12T03:23:34.820 INFO:teuthology.orchestra.run.smithi007.stdout: size: 3 2024-03-12T03:23:34.820 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2024-03-12T03:23:34.820 INFO:teuthology.orchestra.run.smithi007.stdout:service_type: mon 2024-03-12T03:23:34.820 INFO:teuthology.orchestra.run.smithi007.stdout:service_name: mon 2024-03-12T03:23:34.820 INFO:teuthology.orchestra.run.smithi007.stdout:placement: 2024-03-12T03:23:34.820 INFO:teuthology.orchestra.run.smithi007.stdout: count: 3 2024-03-12T03:23:34.820 INFO:teuthology.orchestra.run.smithi007.stdout: hosts: 2024-03-12T03:23:34.821 INFO:teuthology.orchestra.run.smithi007.stdout: - smithi007:172.21.15.7=a 2024-03-12T03:23:34.821 INFO:teuthology.orchestra.run.smithi007.stdout: - smithi033:172.21.15.33=b 2024-03-12T03:23:34.821 INFO:teuthology.orchestra.run.smithi007.stdout: - smithi170:172.21.15.170=c 2024-03-12T03:23:34.821 INFO:teuthology.orchestra.run.smithi007.stdout:status: 2024-03-12T03:23:34.821 INFO:teuthology.orchestra.run.smithi007.stdout: created: '2024-03-12T03:19:56.771772Z' 2024-03-12T03:23:34.821 INFO:teuthology.orchestra.run.smithi007.stdout: last_refresh: '2024-03-12T03:22:14.734403Z' 2024-03-12T03:23:34.821 INFO:teuthology.orchestra.run.smithi007.stdout: running: 3 2024-03-12T03:23:34.821 INFO:teuthology.orchestra.run.smithi007.stdout: size: 3 2024-03-12T03:23:34.821 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2024-03-12T03:23:34.821 INFO:teuthology.orchestra.run.smithi007.stdout:service_type: node-exporter 2024-03-12T03:23:34.821 INFO:teuthology.orchestra.run.smithi007.stdout:service_name: node-exporter 2024-03-12T03:23:34.822 INFO:teuthology.orchestra.run.smithi007.stdout:placement: 2024-03-12T03:23:34.822 INFO:teuthology.orchestra.run.smithi007.stdout: count: 3 2024-03-12T03:23:34.822 INFO:teuthology.orchestra.run.smithi007.stdout: hosts: 2024-03-12T03:23:34.822 INFO:teuthology.orchestra.run.smithi007.stdout: - smithi007=a 2024-03-12T03:23:34.822 INFO:teuthology.orchestra.run.smithi007.stdout: - smithi033=b 2024-03-12T03:23:34.822 INFO:teuthology.orchestra.run.smithi007.stdout: - smithi170=c 2024-03-12T03:23:34.822 INFO:teuthology.orchestra.run.smithi007.stdout:status: 2024-03-12T03:23:34.822 INFO:teuthology.orchestra.run.smithi007.stdout: created: '2024-03-12T03:22:05.919944Z' 2024-03-12T03:23:34.822 INFO:teuthology.orchestra.run.smithi007.stdout: last_refresh: '2024-03-12T03:23:33.997044Z' 2024-03-12T03:23:34.822 INFO:teuthology.orchestra.run.smithi007.stdout: ports: 2024-03-12T03:23:34.822 INFO:teuthology.orchestra.run.smithi007.stdout: - 9100 2024-03-12T03:23:34.823 INFO:teuthology.orchestra.run.smithi007.stdout: running: 2 2024-03-12T03:23:34.823 INFO:teuthology.orchestra.run.smithi007.stdout: size: 3 2024-03-12T03:23:34.823 INFO:teuthology.orchestra.run.smithi007.stdout:events: 2024-03-12T03:23:34.823 INFO:teuthology.orchestra.run.smithi007.stdout:- 2024-03-12T03:23:24.575537Z service:node-exporter [INFO] "service was created" 2024-03-12T03:23:34.823 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2024-03-12T03:23:34.823 INFO:teuthology.orchestra.run.smithi007.stdout:service_type: osd 2024-03-12T03:23:34.824 INFO:teuthology.orchestra.run.smithi007.stdout:service_name: osd 2024-03-12T03:23:34.824 INFO:teuthology.orchestra.run.smithi007.stdout:unmanaged: true 2024-03-12T03:23:34.824 INFO:teuthology.orchestra.run.smithi007.stdout:spec: 2024-03-12T03:23:34.824 INFO:teuthology.orchestra.run.smithi007.stdout: filter_logic: AND 2024-03-12T03:23:34.824 INFO:teuthology.orchestra.run.smithi007.stdout: objectstore: bluestore 2024-03-12T03:23:34.824 INFO:teuthology.orchestra.run.smithi007.stdout:status: 2024-03-12T03:23:34.824 INFO:teuthology.orchestra.run.smithi007.stdout: container_image_id: 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-12T03:23:34.824 INFO:teuthology.orchestra.run.smithi007.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:23:34.824 INFO:teuthology.orchestra.run.smithi007.stdout: last_refresh: '2024-03-12T03:22:14.734568Z' 2024-03-12T03:23:34.824 INFO:teuthology.orchestra.run.smithi007.stdout: running: 3 2024-03-12T03:23:34.824 INFO:teuthology.orchestra.run.smithi007.stdout: size: 3 2024-03-12T03:23:34.824 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2024-03-12T03:23:34.824 INFO:teuthology.orchestra.run.smithi007.stdout:service_type: prometheus 2024-03-12T03:23:34.824 INFO:teuthology.orchestra.run.smithi007.stdout:service_name: prometheus 2024-03-12T03:23:34.825 INFO:teuthology.orchestra.run.smithi007.stdout:placement: 2024-03-12T03:23:34.825 INFO:teuthology.orchestra.run.smithi007.stdout: count: 1 2024-03-12T03:23:34.825 INFO:teuthology.orchestra.run.smithi007.stdout: hosts: 2024-03-12T03:23:34.825 INFO:teuthology.orchestra.run.smithi007.stdout: - smithi033=a 2024-03-12T03:23:34.825 INFO:teuthology.orchestra.run.smithi007.stdout:status: 2024-03-12T03:23:34.825 INFO:teuthology.orchestra.run.smithi007.stdout: created: '2024-03-12T03:22:02.898394Z' 2024-03-12T03:23:34.825 INFO:teuthology.orchestra.run.smithi007.stdout: ports: 2024-03-12T03:23:34.825 INFO:teuthology.orchestra.run.smithi007.stdout: - 9095 2024-03-12T03:23:34.825 INFO:teuthology.orchestra.run.smithi007.stdout: running: 0 2024-03-12T03:23:34.825 INFO:teuthology.orchestra.run.smithi007.stdout: size: 1 2024-03-12T03:23:34.825 INFO:teuthology.orchestra.run.smithi007.stdout:events: 2024-03-12T03:23:34.825 INFO:teuthology.orchestra.run.smithi007.stdout:- 2024-03-12T03:23:24.583251Z service:prometheus [INFO] "service was created" 2024-03-12T03:23:34.825 INFO:teuthology.orchestra.run.smithi007.stdout:--- 2024-03-12T03:23:34.825 INFO:teuthology.orchestra.run.smithi007.stdout:service_type: rgw 2024-03-12T03:23:34.825 INFO:teuthology.orchestra.run.smithi007.stdout:service_id: foo.a 2024-03-12T03:23:34.826 INFO:teuthology.orchestra.run.smithi007.stdout:service_name: rgw.foo.a 2024-03-12T03:23:34.826 INFO:teuthology.orchestra.run.smithi007.stdout:placement: 2024-03-12T03:23:34.826 INFO:teuthology.orchestra.run.smithi007.stdout: count: 1 2024-03-12T03:23:34.826 INFO:teuthology.orchestra.run.smithi007.stdout: hosts: 2024-03-12T03:23:34.826 INFO:teuthology.orchestra.run.smithi007.stdout: - smithi007=foo.a 2024-03-12T03:23:34.826 INFO:teuthology.orchestra.run.smithi007.stdout:status: 2024-03-12T03:23:34.826 INFO:teuthology.orchestra.run.smithi007.stdout: created: '2024-03-12T03:22:03.173151Z' 2024-03-12T03:23:34.826 INFO:teuthology.orchestra.run.smithi007.stdout: last_refresh: '2024-03-12T03:23:30.301618Z' 2024-03-12T03:23:34.826 INFO:teuthology.orchestra.run.smithi007.stdout: ports: 2024-03-12T03:23:34.826 INFO:teuthology.orchestra.run.smithi007.stdout: - 80 2024-03-12T03:23:34.826 INFO:teuthology.orchestra.run.smithi007.stdout: running: 1 2024-03-12T03:23:34.826 INFO:teuthology.orchestra.run.smithi007.stdout: size: 1 2024-03-12T03:23:35.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:35.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:35.098 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:35 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:35.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:35.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:35.263 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:35 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:35.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:35.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:35.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:35 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:35.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-12T03:23:35.941 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:35 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: ::ffff:172.21.15.33 - - [12/Mar/2024:03:23:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-12T03:23:35.948 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:23:36.275 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:36 smithi007 ceph-mon[25800]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:36.275 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:36 smithi007 ceph-mon[25800]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-12T03:23:36.276 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:36 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:36.276 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:36 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:36.276 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:36 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:36.276 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:36 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:36.276 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:36 smithi007 ceph-mon[25800]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:36.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:36 smithi170 ceph-mon[27708]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:36.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:36 smithi170 ceph-mon[27708]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-12T03:23:36.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:36 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:36.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:36 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:36.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:36 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:36.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:36 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:36.364 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:36 smithi170 ceph-mon[27708]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:36.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:36 smithi033 ceph-mon[27552]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:36.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:36 smithi033 ceph-mon[27552]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-12T03:23:36.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:36 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:36.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:36 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:36.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:36 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:36.513 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:36 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:36.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:36 smithi033 ceph-mon[27552]: from='mgr.14448 172.21.15.7:0/517656207' entity='mgr.y' 2024-03-12T03:23:37.895 INFO:teuthology.orchestra.run.smithi007.stdout:osd 3 83s ago - 2024-03-12T03:23:38.184 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:38 smithi007 ceph-mon[25800]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:38.363 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:38 smithi170 ceph-mon[27708]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:38.463 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-12T03:23:38.473 INFO:tasks.cephadm:Teardown begin 2024-03-12T03:23:38.474 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:23:38.505 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:23:38.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:38 smithi033 ceph-mon[27552]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:23:38.560 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:23:38.591 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-12T03:23:38.591 DEBUG:teuthology.orchestra.run.smithi007:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T03:23:38.606 DEBUG:teuthology.orchestra.run.smithi033:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T03:23:38.624 DEBUG:teuthology.orchestra.run.smithi170:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T03:23:38.645 INFO:tasks.cephadm:Stopping all daemons... 2024-03-12T03:23:38.646 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-12T03:23:38.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mon.a 2024-03-12T03:23:38.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:38 smithi007 systemd[1]: Stopping Ceph mon.a for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:23:39.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:38 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-a[25778]: 2024-03-12T03:23:38.979+0000 7f45615e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:23:39.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:38 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-a[25778]: 2024-03-12T03:23:38.979+0000 7f45615e9700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-12T03:23:39.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:39 smithi007 podman[52979]: 2024-03-12 03:23:39.051674245 +0000 UTC m=+0.211911221 container died d8b3346c080857fc0e090db859a5f10822fc26c6ac37c7ce059fb8fd209ce621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-12T03:23:39.305 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:38 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: [12/Mar/2024:03:23:38] ENGINE Bus STOPPING 2024-03-12T03:23:39.305 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:39 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: [12/Mar/2024:03:23:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:23:39.305 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:39 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: [12/Mar/2024:03:23:39] ENGINE Bus STOPPED 2024-03-12T03:23:39.305 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:39 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: [12/Mar/2024:03:23:39] ENGINE Bus STARTING 2024-03-12T03:23:39.305 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:39 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: [12/Mar/2024:03:23:39] ENGINE Serving on http://:::9283 2024-03-12T03:23:39.305 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:39 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: [12/Mar/2024:03:23:39] ENGINE Bus STARTED 2024-03-12T03:23:39.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:39 smithi007 podman[52979]: 2024-03-12 03:23:39.385782787 +0000 UTC m=+0.546019804 container cleanup d8b3346c080857fc0e090db859a5f10822fc26c6ac37c7ce059fb8fd209ce621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_BRANCH=HEAD) 2024-03-12T03:23:39.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:39 smithi007 bash[52979]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-a 2024-03-12T03:23:39.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 12 03:23:39 smithi007 podman[53009]: 2024-03-12 03:23:39.553400266 +0000 UTC m=+0.495505112 container remove d8b3346c080857fc0e090db859a5f10822fc26c6ac37c7ce059fb8fd209ce621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, GIT_BRANCH=HEAD) 2024-03-12T03:23:39.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mon.a.service' 2024-03-12T03:23:39.817 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:23:39.817 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-12T03:23:39.817 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-12T03:23:39.818 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mon.b 2024-03-12T03:23:40.093 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:39 smithi033 systemd[1]: Stopping Ceph mon.b for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:23:41.217 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:41 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-b[27529]: 2024-03-12T03:23:41.026+0000 7ff7d6a83700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:23:41.217 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:41 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-b[27529]: 2024-03-12T03:23:41.026+0000 7ff7d6a83700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-12T03:23:41.763 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:41 smithi033 podman[40852]: 2024-03-12 03:23:41.509624369 +0000 UTC m=+0.609585117 container died 8147033ac0829f010587c741e47ee8c08099300b5da14d3bc3ac478e13b422d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-12T03:23:42.231 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:41 smithi033 podman[40852]: 2024-03-12 03:23:41.87612039 +0000 UTC m=+0.976081146 container cleanup 8147033ac0829f010587c741e47ee8c08099300b5da14d3bc3ac478e13b422d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-12T03:23:42.231 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:41 smithi033 bash[40852]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-b 2024-03-12T03:23:42.231 INFO:journalctl@ceph.mon.b.smithi033.stdout:Mar 12 03:23:42 smithi033 podman[40888]: 2024-03-12 03:23:42.029502515 +0000 UTC m=+0.852543839 container remove 8147033ac0829f010587c741e47ee8c08099300b5da14d3bc3ac478e13b422d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.build-date=20240307) 2024-03-12T03:23:42.252 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mon.b.service' 2024-03-12T03:23:42.299 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:23:42.299 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-12T03:23:42.299 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-12T03:23:42.300 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mon.c 2024-03-12T03:23:42.605 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:42 smithi170 systemd[1]: Stopping Ceph mon.c for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:23:42.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:42 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-c[27686]: 2024-03-12T03:23:42.602+0000 7f246fcb0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:23:42.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:42 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-c[27686]: 2024-03-12T03:23:42.602+0000 7f246fcb0700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-12T03:23:42.863 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:42 smithi170 podman[45641]: 2024-03-12 03:23:42.715377485 +0000 UTC m=+0.215837577 container died a8d1056bc966b66b2aba5b4bd0d409cbdd062c555d8e2d37c3f6975a4c54bb21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-12T03:23:43.353 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:43 smithi170 podman[45641]: 2024-03-12 03:23:43.074374558 +0000 UTC m=+0.574834656 container cleanup a8d1056bc966b66b2aba5b4bd0d409cbdd062c555d8e2d37c3f6975a4c54bb21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, ceph=True) 2024-03-12T03:23:43.353 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 12 03:23:43 smithi170 bash[45641]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mon-c 2024-03-12T03:23:43.467 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mon.c.service' 2024-03-12T03:23:43.520 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:23:43.520 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-12T03:23:43.520 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-12T03:23:43.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mgr.y 2024-03-12T03:23:43.802 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:43 smithi007 systemd[1]: Stopping Ceph mgr.y for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:23:44.098 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:43 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y[26059]: 2024-03-12T03:23:43.799+0000 7f2097f0b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:23:44.098 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:43 smithi007 podman[53122]: 2024-03-12 03:23:43.831627176 +0000 UTC m=+0.115715075 container died 42b9d300140d2fe41159f3b9ea09eaf15cd7f57d7111e46defb8fab00af5d075 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e) 2024-03-12T03:23:44.572 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:44 smithi007 podman[53122]: 2024-03-12 03:23:44.172503098 +0000 UTC m=+0.456591036 container cleanup 42b9d300140d2fe41159f3b9ea09eaf15cd7f57d7111e46defb8fab00af5d075 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-12T03:23:44.592 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:44 smithi007 bash[53122]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y 2024-03-12T03:23:44.592 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Mar 12 03:23:44 smithi007 podman[53142]: 2024-03-12 03:23:44.348696263 +0000 UTC m=+0.518422380 container remove 42b9d300140d2fe41159f3b9ea09eaf15cd7f57d7111e46defb8fab00af5d075 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-y, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:23:44.594 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-12T03:23:44.629 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:23:44.629 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-12T03:23:44.629 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-12T03:23:44.630 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mgr.x 2024-03-12T03:23:44.949 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:23:44 smithi033 systemd[1]: Stopping Ceph mgr.x for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:23:45.209 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:23:44 smithi033 podman[41025]: 2024-03-12 03:23:44.947726322 +0000 UTC m=+0.119199523 container died ec8321b1b7ab2e05a673b2ccde97114ee39cb927cc7c669e2b3a334ff2c9700f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:23:45.513 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:23:45 smithi033 podman[41025]: 2024-03-12 03:23:45.306911183 +0000 UTC m=+0.478384364 container cleanup ec8321b1b7ab2e05a673b2ccde97114ee39cb927cc7c669e2b3a334ff2c9700f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-12T03:23:45.513 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 12 03:23:45 smithi033 bash[41025]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-x 2024-03-12T03:23:45.784 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-12T03:23:45.820 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:23:45.820 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-12T03:23:45.820 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-12T03:23:45.820 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mgr.z 2024-03-12T03:23:46.113 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:23:45 smithi170 systemd[1]: Stopping Ceph mgr.z for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:23:46.447 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:23:46 smithi170 podman[45772]: 2024-03-12 03:23:46.135804259 +0000 UTC m=+0.124883861 container died fe9479a3eaa95670a9da91d153de6b7f217833020408f423f7a78c2036c5b7f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-2b0101e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-12T03:23:46.756 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:23:46 smithi170 podman[45772]: 2024-03-12 03:23:46.528441405 +0000 UTC m=+0.517521005 container cleanup fe9479a3eaa95670a9da91d153de6b7f217833020408f423f7a78c2036c5b7f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-12T03:23:46.756 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 12 03:23:46 smithi170 bash[45772]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-mgr-z 2024-03-12T03:23:46.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@mgr.z.service' 2024-03-12T03:23:46.931 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:23:46.931 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-12T03:23:46.931 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-12T03:23:46.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@osd.0 2024-03-12T03:23:47.230 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:23:46 smithi007 systemd[1]: Stopping Ceph osd.0 for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:23:47.598 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:23:47 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-0[38020]: 2024-03-12T03:23:47.227+0000 7f8bae675700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:23:47.598 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:23:47 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-0[38020]: 2024-03-12T03:23:47.227+0000 7f8bae675700 -1 osd.0 31 *** Got signal Terminated *** 2024-03-12T03:23:47.598 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:23:47 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-0[38020]: 2024-03-12T03:23:47.227+0000 7f8bae675700 -1 osd.0 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:23:52.848 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:23:52 smithi007 podman[53255]: 2024-03-12 03:23:52.629586494 +0000 UTC m=+5.513241387 container died 74794dbb8ec2fe6c16d01845d49b1978fc2dea41b1fa3bb84ae599cc43475ac7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-12T03:23:53.137 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:23:52 smithi007 podman[53255]: 2024-03-12 03:23:52.942741216 +0000 UTC m=+5.826396110 container cleanup 74794dbb8ec2fe6c16d01845d49b1978fc2dea41b1fa3bb84ae599cc43475ac7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e) 2024-03-12T03:23:53.137 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:23:52 smithi007 bash[53255]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-0 2024-03-12T03:23:53.465 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:23:53 smithi007 podman[53469]: 2024-03-12 03:23:53.135723065 +0000 UTC m=+0.511099977 container remove 74794dbb8ec2fe6c16d01845d49b1978fc2dea41b1fa3bb84ae599cc43475ac7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-12T03:23:54.386 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:23:54 smithi007 podman[53653]: 2024-03-12 03:23:54.09828125 +0000 UTC m=+0.106656400 container create 90c32da6a7b72e621176bc68aff592008006237f4a22bd6af024b8da5d1599fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-12T03:23:54.386 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:23:54 smithi007 podman[53653]: 2024-03-12 03:23:54.010767032 +0000 UTC m=+0.019142183 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:23:54.386 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:23:54 smithi007 podman[53653]: 2024-03-12 03:23:54.29242141 +0000 UTC m=+0.300796561 container init 90c32da6a7b72e621176bc68aff592008006237f4a22bd6af024b8da5d1599fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-03-12T03:23:54.386 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:23:54 smithi007 podman[53653]: 2024-03-12 03:23:54.29903371 +0000 UTC m=+0.307408868 container start 90c32da6a7b72e621176bc68aff592008006237f4a22bd6af024b8da5d1599fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-12T03:23:54.387 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:23:54 smithi007 podman[53653]: 2024-03-12 03:23:54.383894934 +0000 UTC m=+0.392270104 container attach 90c32da6a7b72e621176bc68aff592008006237f4a22bd6af024b8da5d1599fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:23:54.922 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 12 03:23:54 smithi007 podman[53653]: 2024-03-12 03:23:54.637924465 +0000 UTC m=+0.646299616 container died 90c32da6a7b72e621176bc68aff592008006237f4a22bd6af024b8da5d1599fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, GIT_BRANCH=HEAD) 2024-03-12T03:23:55.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@osd.0.service' 2024-03-12T03:23:55.296 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:23:55.296 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-12T03:23:55.296 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-12T03:23:55.296 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@osd.1 2024-03-12T03:23:55.613 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:23:55 smithi033 systemd[1]: Stopping Ceph osd.1 for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:23:56.013 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:23:55 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-1[32758]: 2024-03-12T03:23:55.610+0000 7ff7aafb5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:23:56.013 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:23:55 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-1[32758]: 2024-03-12T03:23:55.610+0000 7ff7aafb5700 -1 osd.1 31 *** Got signal Terminated *** 2024-03-12T03:23:56.013 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:23:55 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-1[32758]: 2024-03-12T03:23:55.610+0000 7ff7aafb5700 -1 osd.1 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:24:01.202 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:24:00 smithi033 podman[41883]: 2024-03-12 03:24:00.934473095 +0000 UTC m=+5.435715165 container died 98c4011e73f3be366826204818fbb671cb180b620591d4ffc4cc7529f2618d1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-12T03:24:01.513 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:24:01 smithi033 podman[41883]: 2024-03-12 03:24:01.291206173 +0000 UTC m=+5.792448243 container cleanup 98c4011e73f3be366826204818fbb671cb180b620591d4ffc4cc7529f2618d1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-12T03:24:01.513 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:24:01 smithi033 bash[41883]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-1 2024-03-12T03:24:01.910 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:24:01 smithi033 podman[41905]: 2024-03-12 03:24:01.58386183 +0000 UTC m=+0.647304072 container remove 98c4011e73f3be366826204818fbb671cb180b620591d4ffc4cc7529f2618d1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-1, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-12T03:24:02.221 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:24:01 smithi033 podman[42006]: 2024-03-12 03:24:01.812128831 +0000 UTC m=+0.018379088 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:24:02.222 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:24:01 smithi033 podman[42006]: 2024-03-12 03:24:01.926101289 +0000 UTC m=+0.132351524 container create 80d6037feb4505117363f10f894b3869136e95b8e9cd188cfdc5e18bcb939672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-12T03:24:02.222 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:24:02 smithi033 podman[42006]: 2024-03-12 03:24:02.145232154 +0000 UTC m=+0.351482401 container init 80d6037feb4505117363f10f894b3869136e95b8e9cd188cfdc5e18bcb939672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-12T03:24:02.222 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:24:02 smithi033 podman[42006]: 2024-03-12 03:24:02.150954957 +0000 UTC m=+0.357205193 container start 80d6037feb4505117363f10f894b3869136e95b8e9cd188cfdc5e18bcb939672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-12T03:24:02.222 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:24:02 smithi033 podman[42006]: 2024-03-12 03:24:02.22001652 +0000 UTC m=+0.426266758 container attach 80d6037feb4505117363f10f894b3869136e95b8e9cd188cfdc5e18bcb939672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-2b0101e) 2024-03-12T03:24:02.513 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:24:02 smithi033 podman[42006]: 2024-03-12 03:24:02.445348781 +0000 UTC m=+0.651599010 container died 80d6037feb4505117363f10f894b3869136e95b8e9cd188cfdc5e18bcb939672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-12T03:24:03.042 INFO:journalctl@ceph.osd.1.smithi033.stdout:Mar 12 03:24:03 smithi033 podman[42052]: 2024-03-12 03:24:03.02271142 +0000 UTC m=+0.572449342 container remove 80d6037feb4505117363f10f894b3869136e95b8e9cd188cfdc5e18bcb939672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-12T03:24:03.048 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@osd.1.service' 2024-03-12T03:24:03.091 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:24:03.091 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-12T03:24:03.091 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-12T03:24:03.091 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@osd.2 2024-03-12T03:24:03.373 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:24:03 smithi170 systemd[1]: Stopping Ceph osd.2 for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:24:03.373 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:24:03 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-2[34306]: 2024-03-12T03:24:03.366+0000 7f8fd564a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:24:03.373 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:24:03 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-2[34306]: 2024-03-12T03:24:03.366+0000 7f8fd564a700 -1 osd.2 31 *** Got signal Terminated *** 2024-03-12T03:24:03.373 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:24:03 smithi170 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-2[34306]: 2024-03-12T03:24:03.366+0000 7f8fd564a700 -1 osd.2 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T03:24:09.044 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:24:08 smithi170 podman[46441]: 2024-03-12 03:24:08.746724319 +0000 UTC m=+5.462173741 container died 07edd3bf199d174ca3921f8934fe6ca0f05b1f37c8fe19b5384148ad1f48f4c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:24:09.363 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:24:09 smithi170 podman[46441]: 2024-03-12 03:24:09.19031006 +0000 UTC m=+5.905759473 container cleanup 07edd3bf199d174ca3921f8934fe6ca0f05b1f37c8fe19b5384148ad1f48f4c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-2, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-03-12T03:24:09.363 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:24:09 smithi170 bash[46441]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-2 2024-03-12T03:24:10.457 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:24:10 smithi170 podman[46490]: 2024-03-12 03:24:10.153519967 +0000 UTC m=+1.409155227 container remove 07edd3bf199d174ca3921f8934fe6ca0f05b1f37c8fe19b5384148ad1f48f4c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e) 2024-03-12T03:24:10.708 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:24:10 smithi170 podman[46703]: 2024-03-12 03:24:10.605615745 +0000 UTC m=+0.017581078 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-12T03:24:10.708 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:24:10 smithi170 podman[46703]: 2024-03-12 03:24:10.706724998 +0000 UTC m=+0.118690318 container create aa4524925b50608492716a9d50b14c9e5b7d36f470304c72ea21bdbc6443f994 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-12T03:24:11.045 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:24:10 smithi170 podman[46703]: 2024-03-12 03:24:10.934330979 +0000 UTC m=+0.346296299 container init aa4524925b50608492716a9d50b14c9e5b7d36f470304c72ea21bdbc6443f994 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_CLEAN=True) 2024-03-12T03:24:11.045 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:24:10 smithi170 podman[46703]: 2024-03-12 03:24:10.940423847 +0000 UTC m=+0.352389169 container start aa4524925b50608492716a9d50b14c9e5b7d36f470304c72ea21bdbc6443f994 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-03-12T03:24:11.045 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:24:11 smithi170 podman[46703]: 2024-03-12 03:24:11.04261959 +0000 UTC m=+0.454584916 container attach aa4524925b50608492716a9d50b14c9e5b7d36f470304c72ea21bdbc6443f994 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-12T03:24:11.363 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 12 03:24:11 smithi170 podman[46703]: 2024-03-12 03:24:11.27996579 +0000 UTC m=+0.691931112 container died aa4524925b50608492716a9d50b14c9e5b7d36f470304c72ea21bdbc6443f994 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-12T03:24:11.877 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@osd.2.service' 2024-03-12T03:24:11.921 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:24:11.922 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-12T03:24:11.922 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-12T03:24:11.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-12T03:24:12.241 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:24:11 smithi007 systemd[1]: Stopping Ceph rgw.foo.a for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:24:12.848 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:24:12 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-rgw-foo-a[41272]: 2024-03-12T03:24:12.669+0000 7fed60d70700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:24:12.848 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:24:12 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-rgw-foo-a[41272]: 2024-03-12T03:24:12.669+0000 7fed6f5e8ac0 -1 shutting down 2024-03-12T03:24:23.049 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:24:22 smithi007 bash[54379]: time="2024-03-12T03:24:22Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-12T03:24:23.049 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:24:22 smithi007 podman[54379]: 2024-03-12 03:24:22.696297544 +0000 UTC m=+10.127170989 container died 6018bc187853096a38bd584e6076b5b033ac9bd35976567a0647ec36e42e67c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e) 2024-03-12T03:24:23.348 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:24:23 smithi007 podman[54379]: 2024-03-12 03:24:23.046659023 +0000 UTC m=+10.477532509 container cleanup 6018bc187853096a38bd584e6076b5b033ac9bd35976567a0647ec36e42e67c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-rgw-foo-a, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-12T03:24:23.348 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:24:23 smithi007 bash[54379]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-12T03:24:23.348 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:24:23 smithi007 podman[54857]: 2024-03-12 03:24:23.264704036 +0000 UTC m=+0.562640142 container remove 6018bc187853096a38bd584e6076b5b033ac9bd35976567a0647ec36e42e67c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-12T03:24:23.348 INFO:journalctl@ceph.rgw.foo.a.smithi007.stdout:Mar 12 03:24:23 smithi007 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-12T03:24:23.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-12T03:24:23.527 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:24:23.528 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-12T03:24:23.528 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-12T03:24:23.528 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@prometheus.a 2024-03-12T03:24:23.839 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:23 smithi033 systemd[1]: Stopping Ceph prometheus.a for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:24:24.095 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:23 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:24:23.837Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:24:24.096 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:23 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:24:23.837Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-12T03:24:24.096 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:23 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:24:23.837Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-12T03:24:24.096 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:23 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:24:23.837Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-12T03:24:24.096 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:23 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:24:23.837Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-12T03:24:24.096 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:23 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:24:23.837Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-12T03:24:24.096 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:23 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:24:23.837Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-12T03:24:24.096 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:23 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:24:23.837Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-12T03:24:24.096 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:23 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:24:23.837Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-12T03:24:24.096 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:23 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:24:23.837Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-12T03:24:24.096 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:23 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:24:23.837Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-12T03:24:24.097 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:23 smithi033 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a[35678]: ts=2024-03-12T03:24:23.837Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-12T03:24:24.097 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:23 smithi033 podman[42902]: 2024-03-12 03:24:23.847771296 +0000 UTC m=+0.115119340 container died b307e5aa8866dec1630f66293071bdaa661559ce24cf490d70f4929168f25c50 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:24:24.470 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:24 smithi033 podman[42902]: 2024-03-12 03:24:24.192485903 +0000 UTC m=+0.459833947 container cleanup b307e5aa8866dec1630f66293071bdaa661559ce24cf490d70f4929168f25c50 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:24:24.470 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Mar 12 03:24:24 smithi033 bash[42902]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-prometheus-a 2024-03-12T03:24:24.614 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-12T03:24:24.652 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T03:24:24.652 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-12T03:24:24.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T03:24:24.850 INFO:teuthology.orchestra.run.smithi007.stdout:Deleting cluster with fsid: fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:24:26.098 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:24:25 smithi007 systemd[1]: Stopping Ceph alertmanager.a for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:24:26.571 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:24:26 smithi007 ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a[48326]: ts=2024-03-12T03:24:26.237Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:24:26.571 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:24:26 smithi007 podman[55231]: 2024-03-12 03:24:26.258392086 +0000 UTC m=+0.164698941 container died 47e2fba1cdb1c13d05389a7442d89893f6b33a1c097cbfa9d6072bcaaa25ca19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:24:26.848 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:24:26 smithi007 podman[55231]: 2024-03-12 03:24:26.751743731 +0000 UTC m=+0.658050587 container cleanup 47e2fba1cdb1c13d05389a7442d89893f6b33a1c097cbfa9d6072bcaaa25ca19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:24:26.848 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:24:26 smithi007 bash[55231]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a 2024-03-12T03:24:27.212 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:24:26 smithi007 podman[55251]: 2024-03-12 03:24:26.928042227 +0000 UTC m=+0.668190934 container remove 47e2fba1cdb1c13d05389a7442d89893f6b33a1c097cbfa9d6072bcaaa25ca19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:24:27.212 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:24:27 smithi007 podman[55251]: 2024-03-12 03:24:27.011910638 +0000 UTC m=+0.752059345 volume remove 3cbcb991f80e6e6bc3acfb44b557d912903eb89b2e6d06e017a3d04b62cb77e8 2024-03-12T03:24:27.598 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:24:27 smithi007 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-12T03:24:27.601 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:24:27 smithi007 systemd[1]: Stopped Ceph alertmanager.a for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:24:27.602 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Mar 12 03:24:27 smithi007 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 1.100s CPU time. 2024-03-12T03:24:27.602 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Mar 12 03:24:27 smithi007 systemd[1]: Stopping Ceph node-exporter.a for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:24:28.057 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Mar 12 03:24:27 smithi007 podman[55373]: 2024-03-12 03:24:27.702743084 +0000 UTC m=+0.137568323 container died cd8e0e41303e90c400f0b0bb211758f910b9957c34b63e45036d075ef372eadd (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:24:28.327 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Mar 12 03:24:28 smithi007 podman[55373]: 2024-03-12 03:24:28.197655349 +0000 UTC m=+0.632480628 container cleanup cd8e0e41303e90c400f0b0bb211758f910b9957c34b63e45036d075ef372eadd (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:24:28.327 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Mar 12 03:24:28 smithi007 bash[55373]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-a 2024-03-12T03:24:28.598 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Mar 12 03:24:28 smithi007 podman[55392]: 2024-03-12 03:24:28.407506743 +0000 UTC m=+0.707917305 container remove cd8e0e41303e90c400f0b0bb211758f910b9957c34b63e45036d075ef372eadd (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:24:28.598 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Mar 12 03:24:28 smithi007 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:24:28.598 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Mar 12 03:24:28 smithi007 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-12T03:24:28.598 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Mar 12 03:24:28 smithi007 systemd[1]: Stopped Ceph node-exporter.a for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:24:28.598 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Mar 12 03:24:28 smithi007 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 1.815s CPU time. 2024-03-12T03:24:29.162 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T03:24:29.357 INFO:teuthology.orchestra.run.smithi033.stdout:Deleting cluster with fsid: fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:24:30.513 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:24:30 smithi033 systemd[1]: Stopping Ceph grafana.a for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:24:30.924 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:24:30 smithi033 podman[43280]: 2024-03-12 03:24:30.651410511 +0000 UTC m=+0.114990153 container died a3c1de778e0477eda318c177b977e9ad97ff7a8fbb0dd2e799516c21d822124b (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, description=Ceph Grafana Container, version=8.5, release=239.1651231664, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, distribution-scope=public, vendor=Red Hat, Inc., vcs-type=git) 2024-03-12T03:24:31.263 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:24:31 smithi033 podman[43280]: 2024-03-12 03:24:31.004458751 +0000 UTC m=+0.468038400 container cleanup a3c1de778e0477eda318c177b977e9ad97ff7a8fbb0dd2e799516c21d822124b (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., architecture=x86_64, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-12T03:24:31.263 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:24:31 smithi033 bash[43280]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a 2024-03-12T03:24:31.653 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:24:31 smithi033 podman[43299]: 2024-03-12 03:24:31.288853765 +0000 UTC m=+0.640073485 container remove a3c1de778e0477eda318c177b977e9ad97ff7a8fbb0dd2e799516c21d822124b (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-grafana-a, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., maintainer=Paul Cuzner , io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, vcs-type=git, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, io.buildah.version=1.32.0, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-03-12T03:24:31.653 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:24:31 smithi033 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:24:31.654 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:24:31 smithi033 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-12T03:24:31.654 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:24:31 smithi033 systemd[1]: Stopped Ceph grafana.a for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:24:31.654 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Mar 12 03:24:31 smithi033 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@grafana.a.service: Consumed 4.104s CPU time. 2024-03-12T03:24:32.013 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:24:31 smithi033 systemd[1]: Stopping Ceph node-exporter.b for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:24:32.013 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:24:31 smithi033 podman[43417]: 2024-03-12 03:24:31.88834332 +0000 UTC m=+0.096291473 container died e23ad178c456c6fe3b3aa1faf6275bcf8c31eeb14b79a5f5c53443b0b8751034 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:24:32.401 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:24:32 smithi033 podman[43417]: 2024-03-12 03:24:32.191200665 +0000 UTC m=+0.399148819 container cleanup e23ad178c456c6fe3b3aa1faf6275bcf8c31eeb14b79a5f5c53443b0b8751034 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:24:32.402 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:24:32 smithi033 bash[43417]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b 2024-03-12T03:24:32.716 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:24:32 smithi033 podman[43434]: 2024-03-12 03:24:32.400442282 +0000 UTC m=+0.512736816 container remove e23ad178c456c6fe3b3aa1faf6275bcf8c31eeb14b79a5f5c53443b0b8751034 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:24:32.716 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:24:32 smithi033 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:24:32.716 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:24:32 smithi033 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-12T03:24:32.716 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:24:32 smithi033 systemd[1]: Stopped Ceph node-exporter.b for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:24:32.716 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Mar 12 03:24:32 smithi033 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 1.852s CPU time. 2024-03-12T03:24:33.226 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T03:24:33.421 INFO:teuthology.orchestra.run.smithi170.stdout:Deleting cluster with fsid: fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:24:34.612 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:24:34 smithi170 systemd[1]: Stopping Ceph node-exporter.c for fe67b1e4-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:24:34.863 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:24:34 smithi170 podman[47681]: 2024-03-12 03:24:34.609988824 +0000 UTC m=+0.152926107 container died d913010cc4adffe2ea75ef4b528570a8800bb85c8e8bcd966f1eb72e7c9d58c7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-12T03:24:35.143 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:24:34 smithi170 podman[47681]: 2024-03-12 03:24:34.962590973 +0000 UTC m=+0.505528263 container cleanup d913010cc4adffe2ea75ef4b528570a8800bb85c8e8bcd966f1eb72e7c9d58c7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-12T03:24:35.143 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:24:34 smithi170 bash[47681]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c 2024-03-12T03:24:35.509 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:24:35 smithi170 podman[47699]: 2024-03-12 03:24:35.247342848 +0000 UTC m=+0.640434404 container remove d913010cc4adffe2ea75ef4b528570a8800bb85c8e8bcd966f1eb72e7c9d58c7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-12T03:24:35.509 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:24:35 smithi170 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@node-exporter.c.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:24:35.509 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:24:35 smithi170 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@node-exporter.c.service: Failed with result 'exit-code'. 2024-03-12T03:24:35.509 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:24:35 smithi170 systemd[1]: Stopped Ceph node-exporter.c for fe67b1e4-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:24:35.509 INFO:journalctl@ceph.node-exporter.c.smithi170.stdout:Mar 12 03:24:35 smithi170 systemd[1]: ceph-fe67b1e4-e01e-11ee-95c8-87774f69a715@node-exporter.c.service: Consumed 1.852s CPU time. 2024-03-12T03:24:35.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:24:36.035 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:24:36.069 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:24:36.099 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-12T03:24:36.101 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591867/remote/smithi007/crash 2024-03-12T03:24:36.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/crash -- . 2024-03-12T03:24:36.141 INFO:teuthology.orchestra.run.smithi007.stderr:tar: /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T03:24:36.141 INFO:teuthology.orchestra.run.smithi007.stderr:tar: Error is not recoverable: exiting now 2024-03-12T03:24:36.143 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591867/remote/smithi033/crash 2024-03-12T03:24:36.144 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/crash -- . 2024-03-12T03:24:36.190 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T03:24:36.191 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2024-03-12T03:24:36.192 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591867/remote/smithi170/crash 2024-03-12T03:24:36.193 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/crash -- . 2024-03-12T03:24:36.233 INFO:teuthology.orchestra.run.smithi170.stderr:tar: /var/lib/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T03:24:36.233 INFO:teuthology.orchestra.run.smithi170.stderr:tar: Error is not recoverable: exiting now 2024-03-12T03:24:36.234 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-12T03:24:36.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-12T03:24:36.264 INFO:tasks.cephadm:Compressing logs... 2024-03-12T03:24:36.265 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T03:24:36.307 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T03:24:36.310 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T03:24:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T03:24:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:: No such file or directory 2024-03-12T03:24:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-12T03:24:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T03:24:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-12T03:24:36.333 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mon.a.log: /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-12T03:24:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-12T03:24:36.336 INFO:teuthology.orchestra.run.smithi007.stderr: 57.6% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-12T03:24:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T03:24:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.log 2024-03-12T03:24:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T03:24:36.336 INFO:teuthology.orchestra.run.smithi007.stderr: 89.8% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T03:24:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.log: 82.5% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T03:24:36.336 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-volume.log: 85.1% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T03:24:36.336 INFO:teuthology.orchestra.run.smithi007.stderr: 85.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T03:24:36.341 INFO:teuthology.orchestra.run.smithi033.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-12T03:24:36.342 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T03:24:36.342 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T03:24:36.342 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-12T03:24:36.342 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-12T03:24:36.342 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.audit.log: /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T03:24:36.342 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.log 2024-03-12T03:24:36.342 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mgr.x.log: /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T03:24:36.342 INFO:teuthology.orchestra.run.smithi033.stderr: 81.0% 89.8% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.cephadm.log.gz -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T03:24:36.342 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-12T03:24:36.342 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-12T03:24:36.342 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-volume.log: 84.6% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T03:24:36.342 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mon.b.log: 77.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T03:24:36.343 INFO:teuthology.orchestra.run.smithi170.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T03:24:36.343 INFO:teuthology.orchestra.run.smithi170.stderr:: No such file or directory 2024-03-12T03:24:36.343 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T03:24:36.343 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mgr.z.log 2024-03-12T03:24:36.343 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T03:24:36.343 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.log 2024-03-12T03:24:36.343 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-12T03:24:36.343 INFO:teuthology.orchestra.run.smithi170.stderr: 81.1% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T03:24:36.343 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T03:24:36.344 INFO:teuthology.orchestra.run.smithi170.stderr: 89.9% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T03:24:36.344 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-12T03:24:36.344 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-volume.log: 84.8% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T03:24:36.344 INFO:teuthology.orchestra.run.smithi170.stderr: 80.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T03:24:36.344 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mon.c.log: 90.2% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mgr.z.log.gz 2024-03-12T03:24:36.344 INFO:teuthology.orchestra.run.smithi033.stderr: 90.3% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-12T03:24:36.360 INFO:teuthology.orchestra.run.smithi007.stderr: 90.2% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T03:24:36.362 INFO:teuthology.orchestra.run.smithi033.stderr: 90.2% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T03:24:36.364 INFO:teuthology.orchestra.run.smithi170.stderr: 90.2% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T03:24:36.395 INFO:teuthology.orchestra.run.smithi033.stderr: 92.3% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-12T03:24:36.406 INFO:teuthology.orchestra.run.smithi170.stderr: 92.3% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-12T03:24:36.416 INFO:teuthology.orchestra.run.smithi007.stderr: 88.0% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-12T03:24:36.551 INFO:teuthology.orchestra.run.smithi007.stderr: 91.5% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-12T03:24:36.639 INFO:teuthology.orchestra.run.smithi170.stderr: 94.2% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-12T03:24:36.640 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-12T03:24:36.641 INFO:teuthology.orchestra.run.smithi170.stderr:real 0m0.317s 2024-03-12T03:24:36.641 INFO:teuthology.orchestra.run.smithi170.stderr:user 0m0.404s 2024-03-12T03:24:36.641 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m0.030s 2024-03-12T03:24:36.650 INFO:teuthology.orchestra.run.smithi007.stderr: 94.2% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-12T03:24:36.652 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-12T03:24:36.652 INFO:teuthology.orchestra.run.smithi007.stderr:real 0m0.333s 2024-03-12T03:24:36.652 INFO:teuthology.orchestra.run.smithi007.stderr:user 0m0.641s 2024-03-12T03:24:36.652 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m0.039s 2024-03-12T03:24:36.703 INFO:teuthology.orchestra.run.smithi033.stderr: 94.2% -- replaced with /var/log/ceph/fe67b1e4-e01e-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-12T03:24:36.705 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-12T03:24:36.705 INFO:teuthology.orchestra.run.smithi033.stderr:real 0m0.383s 2024-03-12T03:24:36.705 INFO:teuthology.orchestra.run.smithi033.stderr:user 0m0.455s 2024-03-12T03:24:36.705 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m0.037s 2024-03-12T03:24:36.706 INFO:tasks.cephadm:Archiving logs... 2024-03-12T03:24:36.706 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591867/remote/smithi007/log 2024-03-12T03:24:36.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T03:24:36.843 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591867/remote/smithi033/log 2024-03-12T03:24:36.844 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T03:24:36.939 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591867/remote/smithi170/log 2024-03-12T03:24:36.940 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T03:24:37.033 INFO:tasks.cephadm:Removing cluster... 2024-03-12T03:24:37.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 --force 2024-03-12T03:24:37.219 INFO:teuthology.orchestra.run.smithi007.stdout:Deleting cluster with fsid: fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:24:37.334 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 --force 2024-03-12T03:24:37.523 INFO:teuthology.orchestra.run.smithi033.stdout:Deleting cluster with fsid: fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:24:37.639 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fe67b1e4-e01e-11ee-95c8-87774f69a715 --force 2024-03-12T03:24:37.822 INFO:teuthology.orchestra.run.smithi170.stdout:Deleting cluster with fsid: fe67b1e4-e01e-11ee-95c8-87774f69a715 2024-03-12T03:24:37.936 INFO:tasks.cephadm:Removing cephadm ... 2024-03-12T03:24:37.937 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T03:24:37.954 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T03:24:37.969 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T03:24:37.983 INFO:tasks.cephadm:Teardown complete 2024-03-12T03:24:37.984 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-12T03:24:37.995 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi007:/dev/vg_nvme/lv_1... 2024-03-12T03:24:37.996 DEBUG:teuthology.orchestra.run.smithi007:> sudo nvme disconnect -n lv_1 2024-03-12T03:24:38.144 INFO:teuthology.orchestra.run.smithi007.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-12T03:24:38.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:24:38.146 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi007:/dev/vg_nvme/lv_2... 2024-03-12T03:24:38.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo nvme disconnect -n lv_2 2024-03-12T03:24:38.288 INFO:teuthology.orchestra.run.smithi007.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-12T03:24:38.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:24:38.290 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi007:/dev/vg_nvme/lv_3... 2024-03-12T03:24:38.290 DEBUG:teuthology.orchestra.run.smithi007:> sudo nvme disconnect -n lv_3 2024-03-12T03:24:38.474 INFO:teuthology.orchestra.run.smithi007.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-12T03:24:38.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:24:38.476 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi007:/dev/vg_nvme/lv_4... 2024-03-12T03:24:38.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo nvme disconnect -n lv_4 2024-03-12T03:24:38.650 INFO:teuthology.orchestra.run.smithi007.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-12T03:24:38.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:24:38.652 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-12T03:24:38.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/scratch_devs 2024-03-12T03:24:38.679 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi033:/dev/vg_nvme/lv_1... 2024-03-12T03:24:38.679 DEBUG:teuthology.orchestra.run.smithi033:> sudo nvme disconnect -n lv_1 2024-03-12T03:24:38.823 INFO:teuthology.orchestra.run.smithi033.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-12T03:24:38.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:24:38.825 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi033:/dev/vg_nvme/lv_2... 2024-03-12T03:24:38.825 DEBUG:teuthology.orchestra.run.smithi033:> sudo nvme disconnect -n lv_2 2024-03-12T15:04:14.387 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-03-12T15:04:14.440 DEBUG:teuthology.task.console_log:Killing console logger for smithi007 2024-03-12T15:04:14.442 DEBUG:teuthology.task.console_log:Killing console logger for smithi033 2024-03-12T15:04:14.442 DEBUG:teuthology.task.console_log:Killing console logger for smithi170 2024-03-12T15:04:14.442 DEBUG:teuthology.exit:Finished running handlers