2024-03-11T21:35:57.916 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T21:35:57.916 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928 --owner scheduled_teuthology@teuthology --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} -- /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928/orig.config.yaml 2024-03-11T21:35:57.944 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T21:35:58.019 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928 branch: squid description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591928' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 94a539707733325322964c6a187c6a1b41be1daf ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 94a539707733325322964c6a187c6a1b41be1daf selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 3176 sha1: 94a539707733325322964c6a187c6a1b41be1daf sleep_before_teardown: 0 subset: 14/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 targets: smithi002.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMyy1VnFJ1yzz4Hmpu/kPgfDydhNoluk0E+8C0lV7fgx1Xb657od08p7Wg9G6KJ0QFtpqyx7HLiwEQstPF/jCm4= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOrxgDQKCS1btsbAGFldJn7lmKH+Ph15uGCOQQQh+JkJNirOwLCtFIgLf0iymgkt6lig90ScYBRO3jqYH7njFW4= smithi165.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGPIaeiX8fSXrD2FB9fHS58n7cqRRjH7y6geYjdDoU/1nykpxDVYi6daHwhsPzUUtBxCGd1yWQbz2Ym3OXhoBfI= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T21:35:58.019 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa; will attempt to use it 2024-03-11T21:35:58.020 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks 2024-03-11T21:35:58.020 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T21:35:58.022 INFO:teuthology.task.internal:Checking packages... 2024-03-11T21:35:58.046 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '94a539707733325322964c6a187c6a1b41be1daf' 2024-03-11T21:35:58.046 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T21:35:58.046 INFO:teuthology.packaging:ref: None 2024-03-11T21:35:58.046 INFO:teuthology.packaging:tag: None 2024-03-11T21:35:58.046 INFO:teuthology.packaging:branch: squid 2024-03-11T21:35:58.046 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:35:58.046 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-11T21:35:58.301 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1607.g94a53970 2024-03-11T21:35:58.302 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T21:35:58.308 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T21:35:58.308 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T21:35:58.327 INFO:teuthology.task.internal:Saving configuration 2024-03-11T21:35:58.338 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T21:35:58.344 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:35:58.370 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:30:48.054061', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMyy1VnFJ1yzz4Hmpu/kPgfDydhNoluk0E+8C0lV7fgx1Xb657od08p7Wg9G6KJ0QFtpqyx7HLiwEQstPF/jCm4='} 2024-03-11T21:35:58.394 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:30:48.056839', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOrxgDQKCS1btsbAGFldJn7lmKH+Ph15uGCOQQQh+JkJNirOwLCtFIgLf0iymgkt6lig90ScYBRO3jqYH7njFW4='} 2024-03-11T21:35:58.416 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:30:48.058597', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGPIaeiX8fSXrD2FB9fHS58n7cqRRjH7y6geYjdDoU/1nykpxDVYi6daHwhsPzUUtBxCGd1yWQbz2Ym3OXhoBfI='} 2024-03-11T21:35:58.417 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T21:35:58.423 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-11T21:35:58.423 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-11T21:35:58.423 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-11T21:35:58.423 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T21:35:58.526 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f77afb97f70>, signals=[15]) 2024-03-11T21:35:58.527 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T21:35:58.535 INFO:teuthology.task.internal:Opening connections... 2024-03-11T21:35:58.535 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2024-03-11T21:35:58.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:35:58.643 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-03-11T21:35:58.644 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:35:58.746 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2024-03-11T21:35:58.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:35:58.842 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T21:35:58.849 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2024-03-11T21:35:58.868 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2024-03-11T21:35:58.868 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2024-03-11T21:35:58.926 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="CentOS Stream" 2024-03-11T21:35:58.926 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="9" 2024-03-11T21:35:58.926 INFO:teuthology.orchestra.run.smithi002.stdout:ID="centos" 2024-03-11T21:35:58.926 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="rhel fedora" 2024-03-11T21:35:58.927 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="9" 2024-03-11T21:35:58.927 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el9" 2024-03-11T21:35:58.927 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T21:35:58.927 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2024-03-11T21:35:58.927 INFO:teuthology.orchestra.run.smithi002.stdout:LOGO="fedora-logo-icon" 2024-03-11T21:35:58.927 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T21:35:58.928 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://centos.org/" 2024-03-11T21:35:58.928 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T21:35:58.928 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T21:35:58.928 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T21:35:58.929 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2024-03-11T21:35:58.959 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-03-11T21:35:58.977 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-03-11T21:35:58.977 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-03-11T21:35:59.033 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream" 2024-03-11T21:35:59.033 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="9" 2024-03-11T21:35:59.033 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2024-03-11T21:35:59.033 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2024-03-11T21:35:59.033 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="9" 2024-03-11T21:35:59.034 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el9" 2024-03-11T21:35:59.034 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T21:35:59.034 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2024-03-11T21:35:59.034 INFO:teuthology.orchestra.run.smithi122.stdout:LOGO="fedora-logo-icon" 2024-03-11T21:35:59.034 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T21:35:59.034 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2024-03-11T21:35:59.034 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T21:35:59.034 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T21:35:59.034 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T21:35:59.035 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-03-11T21:35:59.066 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2024-03-11T21:35:59.084 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2024-03-11T21:35:59.084 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2024-03-11T21:35:59.141 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Stream" 2024-03-11T21:35:59.141 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="9" 2024-03-11T21:35:59.141 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos" 2024-03-11T21:35:59.142 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora" 2024-03-11T21:35:59.142 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="9" 2024-03-11T21:35:59.142 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el9" 2024-03-11T21:35:59.142 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T21:35:59.142 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31" 2024-03-11T21:35:59.142 INFO:teuthology.orchestra.run.smithi165.stdout:LOGO="fedora-logo-icon" 2024-03-11T21:35:59.142 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T21:35:59.142 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://centos.org/" 2024-03-11T21:35:59.143 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T21:35:59.143 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T21:35:59.143 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T21:35:59.143 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2024-03-11T21:35:59.174 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T21:35:59.183 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T21:35:59.225 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T21:35:59.226 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2024-03-11T21:35:59.228 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-03-11T21:35:59.230 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2024-03-11T21:35:59.247 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T21:35:59.253 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T21:35:59.253 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2024-03-11T21:35:59.285 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-03-11T21:35:59.287 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2024-03-11T21:35:59.409 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T21:35:59.487 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T21:35:59.502 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T21:35:59.502 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T21:35:59.502 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T21:35:59.503 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T21:35:59.503 DEBUG:teuthology.orchestra.run.smithi165:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T21:35:59.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.520 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-03-11T21:35:59.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.521 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-03-11T21:35:59.522 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-03-11T21:35:59.577 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-361.el9.x86_64 2024-03-11T21:35:59.577 INFO:teuthology.task.kernel:Running kernel on smithi122: 5.14.0-361.el9.x86_64 2024-03-11T21:35:59.577 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum install -y kernel 2024-03-11T21:35:59.579 INFO:teuthology.orchestra.run.smithi165.stdout:5.14.0-361.el9.x86_64 2024-03-11T21:35:59.579 INFO:teuthology.task.kernel:Running kernel on smithi165: 5.14.0-361.el9.x86_64 2024-03-11T21:35:59.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum install -y kernel 2024-03-11T21:35:59.580 INFO:teuthology.orchestra.run.smithi002.stdout:5.14.0-361.el9.x86_64 2024-03-11T21:35:59.580 INFO:teuthology.task.kernel:Running kernel on smithi002: 5.14.0-361.el9.x86_64 2024-03-11T21:35:59.580 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum install -y kernel 2024-03-11T21:36:01.656 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-11T21:36:01.729 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T21:36:01.774 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-11T21:36:03.299 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-11T21:36:03.315 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-11T21:36:03.413 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-11T21:36:04.407 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-11T21:36:06.012 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 3.5 MB/s | 8.0 MB 00:02 2024-03-11T21:36:07.999 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - AppStream 7.7 MB/s | 19 MB 00:02 2024-03-11T21:36:08.680 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-11T21:36:11.462 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 1.0 MB/s | 8.0 MB 00:07 2024-03-11T21:36:14.858 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - CRB 1.9 MB/s | 6.1 MB 00:03 2024-03-11T21:36:16.275 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 15 kB 00:00 2024-03-11T21:36:16.429 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 4.8 MB/s | 19 MB 00:03 2024-03-11T21:36:17.305 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 1.2 MB/s | 6.1 MB 00:05 2024-03-11T21:36:18.748 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 15 kB 00:00 2024-03-11T21:36:20.516 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-11T21:36:20.707 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 7.2 MB/s | 6.1 MB 00:00 2024-03-11T21:36:22.134 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 3.6 MB/s | 21 MB 00:05 2024-03-11T21:36:22.205 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 15 kB 00:00 2024-03-11T21:36:23.073 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 27 MB/s | 21 MB 00:00 2024-03-11T21:36:25.574 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 7.1 kB/s | 1.7 kB 00:00 2024-03-11T21:36:27.209 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-11T21:36:27.756 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T21:36:27.756 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T21:36:27.848 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-11T21:36:27.854 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-11T21:36:27.854 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-03-11T21:36:27.854 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-11T21:36:27.854 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-11T21:36:27.854 INFO:teuthology.orchestra.run.smithi122.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T21:36:27.854 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-11T21:36:27.855 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T21:36:27.855 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T21:36:27.855 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T21:36:27.855 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-11T21:36:27.855 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-11T21:36:27.855 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-11T21:36:27.855 INFO:teuthology.orchestra.run.smithi122.stdout:Install 4 Packages 2024-03-11T21:36:27.855 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-11T21:36:27.856 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 100 M 2024-03-11T21:36:27.856 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 123 M 2024-03-11T21:36:27.856 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-11T21:36:28.089 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-11T21:36:28.709 INFO:teuthology.orchestra.run.smithi122.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 7.4 MB/s | 6.0 MB 00:00 2024-03-11T21:36:29.433 INFO:teuthology.orchestra.run.smithi002.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T21:36:29.433 INFO:teuthology.orchestra.run.smithi002.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T21:36:29.523 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-03-11T21:36:29.530 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-03-11T21:36:29.530 INFO:teuthology.orchestra.run.smithi002.stdout: Package Architecture Version Repository Size 2024-03-11T21:36:29.530 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-03-11T21:36:29.530 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-03-11T21:36:29.530 INFO:teuthology.orchestra.run.smithi002.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T21:36:29.530 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-03-11T21:36:29.530 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T21:36:29.530 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T21:36:29.530 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T21:36:29.531 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:36:29.531 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-03-11T21:36:29.531 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-03-11T21:36:29.531 INFO:teuthology.orchestra.run.smithi002.stdout:Install 4 Packages 2024-03-11T21:36:29.531 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:36:29.531 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 100 M 2024-03-11T21:36:29.531 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 123 M 2024-03-11T21:36:29.532 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-03-11T21:36:29.643 INFO:teuthology.orchestra.run.smithi122.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 12 MB/s | 21 MB 00:01 2024-03-11T21:36:30.246 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T21:36:30.246 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T21:36:30.342 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-03-11T21:36:30.343 INFO:teuthology.orchestra.run.smithi122.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 16 MB/s | 39 MB 00:02 2024-03-11T21:36:30.354 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-03-11T21:36:30.354 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-03-11T21:36:30.354 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-03-11T21:36:30.355 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-03-11T21:36:30.355 INFO:teuthology.orchestra.run.smithi165.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T21:36:30.355 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-03-11T21:36:30.355 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T21:36:30.355 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T21:36:30.355 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T21:36:30.355 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T21:36:30.355 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-03-11T21:36:30.355 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-03-11T21:36:30.355 INFO:teuthology.orchestra.run.smithi165.stdout:Install 4 Packages 2024-03-11T21:36:30.355 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T21:36:30.356 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 100 M 2024-03-11T21:36:30.356 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 123 M 2024-03-11T21:36:30.356 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-03-11T21:36:30.726 INFO:teuthology.orchestra.run.smithi122.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 17 MB/s | 33 MB 00:02 2024-03-11T21:36:30.727 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:36:30.727 INFO:teuthology.orchestra.run.smithi122.stdout:Total 35 MB/s | 100 MB 00:02 2024-03-11T21:36:30.757 INFO:teuthology.orchestra.run.smithi165.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 22 MB/s | 6.0 MB 00:00 2024-03-11T21:36:31.091 INFO:teuthology.orchestra.run.smithi165.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 34 MB/s | 21 MB 00:00 2024-03-11T21:36:31.613 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-11T21:36:31.641 INFO:teuthology.orchestra.run.smithi165.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 34 MB/s | 39 MB 00:01 2024-03-11T21:36:31.925 INFO:teuthology.orchestra.run.smithi165.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 29 MB/s | 33 MB 00:01 2024-03-11T21:36:31.925 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:36:31.925 INFO:teuthology.orchestra.run.smithi165.stdout:Total 63 MB/s | 100 MB 00:01 2024-03-11T21:36:31.981 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-11T21:36:31.982 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-11T21:36:32.502 INFO:teuthology.orchestra.run.smithi002.stdout:(1/4): kernel-core-5.14.0-427.el9.x86_64.rpm 7.2 MB/s | 21 MB 00:02 2024-03-11T21:36:32.809 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-03-11T21:36:33.197 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-03-11T21:36:33.197 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-03-11T21:36:33.210 INFO:teuthology.orchestra.run.smithi002.stdout:(2/4): kernel-5.14.0-427.el9.x86_64.rpm 1.7 MB/s | 6.0 MB 00:03 2024-03-11T21:36:33.618 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-11T21:36:33.619 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-11T21:36:34.566 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-03-11T21:36:34.566 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-03-11T21:36:36.554 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-11T21:36:37.217 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:36:37.315 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-03-11T21:36:37.363 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:36:37.969 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:36:38.110 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:36:40.118 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:36:40.303 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:36:40.900 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:36:41.084 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:36:45.142 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:36:45.356 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:36:45.824 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:36:45.943 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:36:50.137 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:36:50.736 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:37:07.515 INFO:teuthology.orchestra.run.smithi002.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 977 kB/s | 33 MB 00:35 2024-03-11T21:37:14.615 INFO:teuthology.orchestra.run.smithi002.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 894 kB/s | 39 MB 00:45 2024-03-11T21:37:14.615 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:37:14.615 INFO:teuthology.orchestra.run.smithi002.stdout:Total 2.2 MB/s | 100 MB 00:45 2024-03-11T21:37:15.530 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-03-11T21:37:15.966 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-03-11T21:37:15.967 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-03-11T21:37:17.359 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-03-11T21:37:17.359 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-03-11T21:37:19.173 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:37:19.174 INFO:teuthology.orchestra.run.smithi122.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T21:37:19.174 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-11T21:37:19.259 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:37:19.746 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:37:19.747 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:37:19.747 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:37:19.747 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:37:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:37:19.990 INFO:teuthology.orchestra.run.smithi165.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T21:37:19.990 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T21:37:20.074 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:37:20.194 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-03-11T21:37:20.201 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:37:20.201 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-11T21:37:20.201 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-11T21:37:20.201 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T21:37:20.201 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T21:37:20.201 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T21:37:20.201 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T21:37:20.201 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-11T21:37:20.201 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-11T21:37:20.611 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:37:20.611 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:37:20.611 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:37:20.611 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:37:20.712 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:37:20.856 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:37:20.856 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:37:20.857 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T21:37:20.857 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T21:37:20.857 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T21:37:20.857 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-11T21:37:20.857 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T21:37:20.857 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:37:20.871 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:37:20.978 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:37:20.978 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:37:20.978 DEBUG:teuthology.orchestra.run.smithi122:> sudo rpm -qi grub2-tools 2024-03-11T21:37:21.006 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:37:21.020 INFO:teuthology.orchestra.run.smithi122.stdout:Name : grub2-tools 2024-03-11T21:37:21.020 INFO:teuthology.orchestra.run.smithi122.stdout:Epoch : 1 2024-03-11T21:37:21.020 INFO:teuthology.orchestra.run.smithi122.stdout:Version : 2.06 2024-03-11T21:37:21.020 INFO:teuthology.orchestra.run.smithi122.stdout:Release : 68.el9 2024-03-11T21:37:21.020 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: x86_64 2024-03-11T21:37:21.020 INFO:teuthology.orchestra.run.smithi122.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T21:37:21.020 INFO:teuthology.orchestra.run.smithi122.stdout:Group : Unspecified 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout:Size : 8264254 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout:License : GPLv3+ 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout:Packager : builder@centos.org 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout:Vendor : CentOS 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout:Summary : Support tools for GRUB. 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout:Description : 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T21:37:21.021 INFO:teuthology.orchestra.run.smithi122.stdout:hardware devices. 2024-03-11T21:37:21.022 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-11T21:37:21.022 INFO:teuthology.orchestra.run.smithi122.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T21:37:21.022 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T21:37:21.023 INFO:teuthology.task.kernel:Updating grub on smithi122 to boot 5.14.0-427.el9.x86_64 2024-03-11T21:37:21.023 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T21:37:21.064 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:37:21.064 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T21:37:21.064 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-03-11T21:37:21.064 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T21:37:21.064 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T21:37:21.064 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T21:37:21.064 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T21:37:21.064 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T21:37:21.065 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-03-11T21:37:21.597 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2024-03-11T21:37:21.598 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:37:21.784 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:37:21.784 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:37:21.784 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T21:37:21.784 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T21:37:21.784 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T21:37:21.784 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-11T21:37:21.784 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi165.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T21:37:21.785 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:37:21.906 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:37:21.906 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:37:21.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo rpm -qi grub2-tools 2024-03-11T21:37:21.940 INFO:teuthology.orchestra.run.smithi165.stdout:Name : grub2-tools 2024-03-11T21:37:21.940 INFO:teuthology.orchestra.run.smithi165.stdout:Epoch : 1 2024-03-11T21:37:21.940 INFO:teuthology.orchestra.run.smithi165.stdout:Version : 2.06 2024-03-11T21:37:21.940 INFO:teuthology.orchestra.run.smithi165.stdout:Release : 68.el9 2024-03-11T21:37:21.940 INFO:teuthology.orchestra.run.smithi165.stdout:Architecture: x86_64 2024-03-11T21:37:21.941 INFO:teuthology.orchestra.run.smithi165.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T21:37:21.941 INFO:teuthology.orchestra.run.smithi165.stdout:Group : Unspecified 2024-03-11T21:37:21.941 INFO:teuthology.orchestra.run.smithi165.stdout:Size : 8264254 2024-03-11T21:37:21.941 INFO:teuthology.orchestra.run.smithi165.stdout:License : GPLv3+ 2024-03-11T21:37:21.941 INFO:teuthology.orchestra.run.smithi165.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T21:37:21.941 INFO:teuthology.orchestra.run.smithi165.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T21:37:21.941 INFO:teuthology.orchestra.run.smithi165.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T21:37:21.941 INFO:teuthology.orchestra.run.smithi165.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T21:37:21.941 INFO:teuthology.orchestra.run.smithi165.stdout:Packager : builder@centos.org 2024-03-11T21:37:21.941 INFO:teuthology.orchestra.run.smithi165.stdout:Vendor : CentOS 2024-03-11T21:37:21.941 INFO:teuthology.orchestra.run.smithi165.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T21:37:21.941 INFO:teuthology.orchestra.run.smithi165.stdout:Summary : Support tools for GRUB. 2024-03-11T21:37:21.941 INFO:teuthology.orchestra.run.smithi165.stdout:Description : 2024-03-11T21:37:21.941 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T21:37:21.941 INFO:teuthology.orchestra.run.smithi165.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T21:37:21.942 INFO:teuthology.orchestra.run.smithi165.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T21:37:21.942 INFO:teuthology.orchestra.run.smithi165.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T21:37:21.942 INFO:teuthology.orchestra.run.smithi165.stdout:hardware devices. 2024-03-11T21:37:21.942 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T21:37:21.942 INFO:teuthology.orchestra.run.smithi165.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T21:37:21.943 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T21:37:21.943 INFO:teuthology.task.kernel:Updating grub on smithi165 to boot 5.14.0-427.el9.x86_64 2024-03-11T21:37:21.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T21:37:22.575 INFO:teuthology.orchestra.run.smithi165.stderr:Generating grub configuration file ... 2024-03-11T21:37:22.984 INFO:teuthology.orchestra.run.smithi122.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T21:37:23.011 INFO:teuthology.orchestra.run.smithi122.stderr:done 2024-03-11T21:37:23.013 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-03-11T21:37:23.028 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.VyLrH0DZBM 2024-03-11T21:37:23.028 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VyLrH0DZBM 2024-03-11T21:37:23.102 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.VyLrH0DZBM 2024-03-11T21:37:23.258 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.VyLrH0DZBM is 6KB 2024-03-11T21:37:23.307 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.VyLrH0DZBM 2024-03-11T21:37:23.321 DEBUG:teuthology.orchestra.run.smithi122:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T21:37:23.389 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T21:37:23.389 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T21:37:23.389 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T21:37:23.390 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T21:37:23.391 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T21:37:23.532 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2024-03-11T21:37:23.904 INFO:teuthology.orchestra.run.smithi165.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T21:37:23.918 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:37:23.936 INFO:teuthology.orchestra.run.smithi165.stderr:done 2024-03-11T21:37:23.938 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-03-11T21:37:23.953 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.h0d1EoNfvl 2024-03-11T21:37:23.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.h0d1EoNfvl 2024-03-11T21:37:24.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.h0d1EoNfvl 2024-03-11T21:37:24.101 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:37:24.158 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.h0d1EoNfvl is 6KB 2024-03-11T21:37:24.207 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.h0d1EoNfvl 2024-03-11T21:37:24.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T21:37:24.286 INFO:teuthology.orchestra.run.smithi165.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T21:37:24.287 INFO:teuthology.orchestra.run.smithi165.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T21:37:24.287 INFO:teuthology.orchestra.run.smithi165.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T21:37:24.287 INFO:teuthology.orchestra.run.smithi165.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T21:37:24.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T21:37:24.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo shutdown -r now 2024-03-11T21:37:29.190 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:37:29.313 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:37:34.432 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:37:53.535 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-11T21:37:53.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-11T21:37:53.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:37:54.448 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-11T21:37:54.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-11T21:37:54.449 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:38:05.393 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:38:05.393 INFO:teuthology.orchestra.run.smithi002.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T21:38:05.393 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:38:05.479 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:38:05.947 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:38:05.947 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:38:05.947 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:38:05.947 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:38:06.380 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:38:06.380 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:38:06.380 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-03-11T21:38:06.380 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T21:38:06.381 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T21:38:06.381 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T21:38:06.381 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T21:38:06.381 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:38:06.381 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-03-11T21:38:06.884 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:38:07.034 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:38:07.035 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:38:07.036 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T21:38:07.036 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T21:38:07.036 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T21:38:07.036 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-11T21:38:07.036 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T21:38:07.036 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:38:07.162 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:38:07.163 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:38:07.163 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -qi grub2-tools 2024-03-11T21:38:07.212 INFO:teuthology.orchestra.run.smithi002.stdout:Name : grub2-tools 2024-03-11T21:38:07.212 INFO:teuthology.orchestra.run.smithi002.stdout:Epoch : 1 2024-03-11T21:38:07.212 INFO:teuthology.orchestra.run.smithi002.stdout:Version : 2.06 2024-03-11T21:38:07.212 INFO:teuthology.orchestra.run.smithi002.stdout:Release : 68.el9 2024-03-11T21:38:07.212 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: x86_64 2024-03-11T21:38:07.212 INFO:teuthology.orchestra.run.smithi002.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T21:38:07.212 INFO:teuthology.orchestra.run.smithi002.stdout:Group : Unspecified 2024-03-11T21:38:07.212 INFO:teuthology.orchestra.run.smithi002.stdout:Size : 8264254 2024-03-11T21:38:07.212 INFO:teuthology.orchestra.run.smithi002.stdout:License : GPLv3+ 2024-03-11T21:38:07.212 INFO:teuthology.orchestra.run.smithi002.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T21:38:07.212 INFO:teuthology.orchestra.run.smithi002.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T21:38:07.212 INFO:teuthology.orchestra.run.smithi002.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T21:38:07.212 INFO:teuthology.orchestra.run.smithi002.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T21:38:07.213 INFO:teuthology.orchestra.run.smithi002.stdout:Packager : builder@centos.org 2024-03-11T21:38:07.213 INFO:teuthology.orchestra.run.smithi002.stdout:Vendor : CentOS 2024-03-11T21:38:07.213 INFO:teuthology.orchestra.run.smithi002.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T21:38:07.213 INFO:teuthology.orchestra.run.smithi002.stdout:Summary : Support tools for GRUB. 2024-03-11T21:38:07.213 INFO:teuthology.orchestra.run.smithi002.stdout:Description : 2024-03-11T21:38:07.213 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:38:07.213 INFO:teuthology.orchestra.run.smithi002.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T21:38:07.213 INFO:teuthology.orchestra.run.smithi002.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T21:38:07.213 INFO:teuthology.orchestra.run.smithi002.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T21:38:07.213 INFO:teuthology.orchestra.run.smithi002.stdout:hardware devices. 2024-03-11T21:38:07.213 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:38:07.213 INFO:teuthology.orchestra.run.smithi002.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T21:38:07.214 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T21:38:07.214 INFO:teuthology.task.kernel:Updating grub on smithi002 to boot 5.14.0-427.el9.x86_64 2024-03-11T21:38:07.214 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T21:38:07.791 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2024-03-11T21:38:09.183 INFO:teuthology.orchestra.run.smithi002.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T21:38:09.214 INFO:teuthology.orchestra.run.smithi002.stderr:done 2024-03-11T21:38:09.217 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2024-03-11T21:38:09.232 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.i9vLIFnSuT 2024-03-11T21:38:09.232 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.i9vLIFnSuT 2024-03-11T21:38:09.304 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.i9vLIFnSuT 2024-03-11T21:38:09.450 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.i9vLIFnSuT is 6KB 2024-03-11T21:38:09.499 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.i9vLIFnSuT 2024-03-11T21:38:09.513 DEBUG:teuthology.orchestra.run.smithi002:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T21:38:09.589 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T21:38:09.590 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T21:38:09.590 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T21:38:09.590 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T21:38:09.591 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T21:38:09.732 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2024-03-11T21:38:27.999 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-03-11T21:38:29.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-03-11T21:38:37.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-11T21:38:37.006 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:38:38.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-11T21:38:38.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:38:39.737 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-11T21:38:39.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-03-11T21:38:39.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:38:55.394 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-03-11T21:38:56.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-03-11T21:39:07.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-11T21:39:07.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:39:07.713 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-03-11T21:39:08.166 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-11T21:39:08.167 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-11T21:39:08.167 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-03-11T21:39:08.180 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-427.el9.x86_64 2024-03-11T21:39:08.181 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T21:39:08.181 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:39:08.181 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T21:39:08.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-11T21:39:08.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:39:09.182 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-11T21:39:09.182 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:39:09.243 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-03-11T21:39:09.279 DEBUG:teuthology.parallel:result is None 2024-03-11T21:39:11.221 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-03-11T21:39:11.734 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-11T21:39:11.734 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-427.el9.x86_64"... 2024-03-11T21:39:11.735 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-03-11T21:39:11.749 INFO:teuthology.orchestra.run.smithi165.stdout:5.14.0-427.el9.x86_64 2024-03-11T21:39:11.749 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T21:39:11.750 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:39:11.750 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T21:39:12.751 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-11T21:39:12.751 DEBUG:teuthology.orchestra.run.smithi165:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:39:12.846 INFO:teuthology.orchestra.run.smithi165.stdout:ttyS1 2024-03-11T21:39:12.879 DEBUG:teuthology.parallel:result is None 2024-03-11T21:39:39.740 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T21:39:48.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-03-11T21:39:48.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:39:49.100 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-03-11T21:39:49.501 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-03-11T21:39:49.501 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T21:39:49.501 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-03-11T21:39:49.515 INFO:teuthology.orchestra.run.smithi002.stdout:5.14.0-427.el9.x86_64 2024-03-11T21:39:49.515 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T21:39:49.515 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:39:49.515 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T21:39:50.516 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-11T21:39:50.516 DEBUG:teuthology.orchestra.run.smithi002:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:39:50.579 INFO:teuthology.orchestra.run.smithi002.stdout:ttyS1 2024-03-11T21:39:50.629 DEBUG:teuthology.parallel:result is None 2024-03-11T21:39:50.630 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T21:39:50.637 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T21:39:50.637 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:39:50.639 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:39:50.642 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:39:50.662 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T21:39:50.668 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T21:39:50.674 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T21:39:50.675 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:39:50.691 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:39:50.700 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:39:50.730 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T21:39:50.736 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T21:39:50.737 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:39:50.743 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:39:50.762 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:39:50.778 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:39:50.792 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:39:50.794 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:39:50.807 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:39:50.808 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:39:50.821 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:39:50.823 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T21:39:50.831 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T21:39:50.831 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:39:50.833 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:39:50.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:39:50.893 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T21:39:50.903 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T21:39:50.903 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:39:50.905 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:39:50.921 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:39:50.950 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:39:50.997 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:39:51.039 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:39:51.116 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:39:51.155 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T21:39:51.155 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:39:51.219 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:39:51.261 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:39:51.310 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:39:51.394 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:39:51.435 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-11T21:39:51.435 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:39:51.502 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:39:51.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:39:51.608 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:39:51.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:39:51.696 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T21:39:51.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:39:51.765 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2024-03-11T21:39:51.767 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-03-11T21:39:51.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2024-03-11T21:39:51.809 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:39:51.821 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:39:51.866 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:39:52.212 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T21:39:52.220 INFO:teuthology.task.internal:Starting timer... 2024-03-11T21:39:52.220 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T21:39:52.234 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T21:39:52.243 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-11T21:39:52.244 DEBUG:teuthology.orchestra.run.smithi002:> sudo service auditd rotate 2024-03-11T21:39:52.352 INFO:teuthology.orchestra.run.smithi002.stdout:Rotating logs: 2024-03-11T21:39:52.357 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2024-03-11T21:39:52.416 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2024-03-11T21:39:52.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo service auditd rotate 2024-03-11T21:39:52.519 INFO:teuthology.orchestra.run.smithi165.stdout:Rotating logs: 2024-03-11T21:39:52.522 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T21:39:52.522 DEBUG:teuthology.orchestra.run.smithi002:> /usr/sbin/getenforce 2024-03-11T21:39:52.556 INFO:teuthology.orchestra.run.smithi002.stdout:Permissive 2024-03-11T21:39:52.556 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2024-03-11T21:39:52.577 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2024-03-11T21:39:52.577 DEBUG:teuthology.orchestra.run.smithi165:> /usr/sbin/getenforce 2024-03-11T21:39:52.604 INFO:teuthology.orchestra.run.smithi165.stdout:Permissive 2024-03-11T21:39:52.604 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi002.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi122.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi165.front.sepia.ceph.com': 'permissive'} 2024-03-11T21:39:52.604 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T21:39:52.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.650 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T21:39:52.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T21:39:52.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.709 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T21:39:52.709 DEBUG:teuthology.orchestra.run.smithi002:> sudo /usr/sbin/setenforce permissive 2024-03-11T21:39:53.051 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2024-03-11T21:39:53.095 DEBUG:teuthology.orchestra.run.smithi165:> sudo /usr/sbin/setenforce permissive 2024-03-11T21:39:53.130 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T21:39:53.139 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T21:39:53.156 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T21:39:53.158 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com,smithi122.front.sepia.ceph.com,smithi165.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T21:44:08.071 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com'), Remote(name='ubuntu@smithi165.front.sepia.ceph.com')] 2024-03-11T21:44:08.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-03-11T21:44:08.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:44:08.163 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-03-11T21:44:08.240 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-03-11T21:44:08.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-11T21:44:08.242 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:44:08.328 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-03-11T21:44:08.404 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-11T21:44:08.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-11T21:44:08.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:44:08.492 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-03-11T21:44:08.574 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-11T21:44:08.575 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T21:44:08.585 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T21:44:08.585 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:44:08.586 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:44:08.588 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:44:08.588 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:44:08.590 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:44:08.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:44:08.618 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T21:44:08.620 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T21:44:08.636 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T21:44:08.637 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T21:44:08.648 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T21:44:08.666 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T21:44:08.666 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2024-03-11T21:44:08.666 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: ntpd: command not found 2024-03-11T21:44:08.679 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2024-03-11T21:44:08.680 INFO:teuthology.orchestra.run.smithi002.stdout:506 Cannot talk to daemon 2024-03-11T21:44:08.695 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T21:44:08.696 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: ntpd: command not found 2024-03-11T21:44:08.696 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T21:44:08.710 INFO:teuthology.orchestra.run.smithi165.stdout:506 Cannot talk to daemon 2024-03-11T21:44:08.712 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T21:44:08.713 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T21:44:08.727 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T21:44:08.743 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T21:44:08.768 INFO:teuthology.orchestra.run.smithi002.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:44:08.768 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:44:08.772 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:44:08.772 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2024-03-11T21:44:08.772 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:44:08.772 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:44:08.772 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:44:08.772 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:44:08.773 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:44:08.773 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-03-11T21:44:08.773 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:44:08.773 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:44:08.773 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:44:08.773 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:44:08.802 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:44:08.805 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:44:08.806 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-03-11T21:44:08.806 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:44:08.806 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:44:08.806 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:44:08.806 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:44:08.806 INFO:teuthology.run_tasks:Running task pexec... 2024-03-11T21:44:08.815 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-11T21:44:08.816 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi002.front.sepia.ceph.com 2024-03-11T21:44:08.816 DEBUG:teuthology.orchestra.run.smithi002:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T21:44:08.816 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi122.front.sepia.ceph.com 2024-03-11T21:44:08.816 DEBUG:teuthology.orchestra.run.smithi122:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T21:44:08.817 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi165.front.sepia.ceph.com 2024-03-11T21:44:08.817 DEBUG:teuthology.orchestra.run.smithi165:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T21:44:09.394 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:51 ago on Mon 11 Mar 2024 09:43:18 PM UTC. 2024-03-11T21:44:09.405 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:49 ago on Mon 11 Mar 2024 09:43:20 PM UTC. 2024-03-11T21:44:09.427 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:51 ago on Mon 11 Mar 2024 09:43:18 PM UTC. 2024-03-11T21:44:09.540 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-11T21:44:09.553 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-11T21:44:09.553 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-03-11T21:44:09.553 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-11T21:44:09.553 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-11T21:44:09.553 INFO:teuthology.orchestra.run.smithi122.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T21:44:09.553 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-11T21:44:09.553 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-11T21:44:09.553 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-11T21:44:09.554 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-11T21:44:09.554 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-11T21:44:09.556 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-03-11T21:44:09.558 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-03-11T21:44:09.558 INFO:teuthology.orchestra.run.smithi002.stdout: Package Architecture Version Repository Size 2024-03-11T21:44:09.558 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-03-11T21:44:09.558 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-03-11T21:44:09.558 INFO:teuthology.orchestra.run.smithi002.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T21:44:09.558 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:44:09.558 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-03-11T21:44:09.558 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-03-11T21:44:09.558 INFO:teuthology.orchestra.run.smithi002.stdout:Install 1 Package 2024-03-11T21:44:09.558 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:44:09.563 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 3.1 M 2024-03-11T21:44:09.564 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 9.8 M 2024-03-11T21:44:09.564 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-11T21:44:09.574 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-03-11T21:44:09.577 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 3.1 M 2024-03-11T21:44:09.577 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 9.8 M 2024-03-11T21:44:09.577 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-03-11T21:44:09.584 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-03-11T21:44:09.584 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-03-11T21:44:09.585 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-03-11T21:44:09.585 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-03-11T21:44:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T21:44:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T21:44:09.585 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-03-11T21:44:09.585 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-03-11T21:44:09.585 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-03-11T21:44:09.585 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T21:44:09.588 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 3.1 M 2024-03-11T21:44:09.588 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 9.8 M 2024-03-11T21:44:09.588 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-03-11T21:44:09.915 INFO:teuthology.orchestra.run.smithi002.stdout:runc-1.1.12-2.el9.x86_64.rpm 11 MB/s | 3.1 MB 00:00 2024-03-11T21:44:09.915 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:44:09.915 INFO:teuthology.orchestra.run.smithi002.stdout:Total 9.3 MB/s | 3.1 MB 00:00 2024-03-11T21:44:09.951 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-03-11T21:44:09.957 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-03-11T21:44:09.957 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-03-11T21:44:09.987 INFO:teuthology.orchestra.run.smithi165.stdout:runc-1.1.12-2.el9.x86_64.rpm 8.8 MB/s | 3.1 MB 00:00 2024-03-11T21:44:09.988 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:44:09.988 INFO:teuthology.orchestra.run.smithi165.stdout:Total 7.8 MB/s | 3.1 MB 00:00 2024-03-11T21:44:10.024 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-03-11T21:44:10.030 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-03-11T21:44:10.030 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-03-11T21:44:10.039 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-03-11T21:44:10.040 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-03-11T21:44:10.109 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-03-11T21:44:10.109 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-03-11T21:44:10.259 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-03-11T21:44:10.336 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:44:10.337 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-03-11T21:44:10.432 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:44:10.898 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:44:11.047 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:44:11.252 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:44:11.252 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:44:11.252 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-03-11T21:44:11.252 INFO:teuthology.orchestra.run.smithi002.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T21:44:11.252 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:44:11.252 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-03-11T21:44:11.383 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:44:11.383 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T21:44:11.383 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-03-11T21:44:11.383 INFO:teuthology.orchestra.run.smithi165.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T21:44:11.383 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T21:44:11.383 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-03-11T21:44:11.535 DEBUG:teuthology.parallel:result is None 2024-03-11T21:44:11.669 DEBUG:teuthology.parallel:result is None 2024-03-11T21:44:14.773 INFO:teuthology.orchestra.run.smithi122.stdout:runc-1.1.12-2.el9.x86_64.rpm 620 kB/s | 3.1 MB 00:05 2024-03-11T21:44:14.773 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:44:14.774 INFO:teuthology.orchestra.run.smithi122.stdout:Total 615 kB/s | 3.1 MB 00:05 2024-03-11T21:44:14.812 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-11T21:44:14.818 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-11T21:44:14.818 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-11T21:44:14.898 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-11T21:44:14.899 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-11T21:44:15.118 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-11T21:44:15.201 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:44:15.727 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:44:16.031 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:44:16.031 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-11T21:44:16.031 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-11T21:44:16.031 INFO:teuthology.orchestra.run.smithi122.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T21:44:16.031 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-11T21:44:16.031 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-11T21:44:16.311 DEBUG:teuthology.parallel:result is None 2024-03-11T21:44:16.311 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-11T21:44:16.322 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-11T21:44:16.322 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T21:44:16.322 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:44:16.340 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T21:44:16.341 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 2024-03-11T21:44:16.398 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T21:44:16.398 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:44:16.398 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T21:44:16.398 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:44:16.398 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:44:16.399 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-03-11 21:44:07.088429694 +0000 2024-03-11T21:44:16.399 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-03-11 21:43:43.921947122 +0000 2024-03-11T21:44:16.399 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-03-11 21:43:43.921947122 +0000 2024-03-11T21:44:16.399 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-03-11 21:43:43.921947122 +0000 2024-03-11T21:44:16.399 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T21:44:16.467 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-03-11T21:44:16.468 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-03-11T21:44:16.468 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000112644 s, 4.5 MB/s 2024-03-11T21:44:16.469 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T21:44:16.527 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 2024-03-11T21:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T21:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-11T21:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-03-11 21:44:07.088429694 +0000 2024-03-11T21:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-03-11 21:43:44.258939596 +0000 2024-03-11T21:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-03-11 21:43:44.258939596 +0000 2024-03-11T21:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-03-11 21:43:44.258939596 +0000 2024-03-11T21:44:16.584 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T21:44:16.652 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-03-11T21:44:16.652 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-03-11T21:44:16.652 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 9.958e-05 s, 5.1 MB/s 2024-03-11T21:44:16.653 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T21:44:16.710 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 2024-03-11T21:44:16.766 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T21:44:16.766 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:44:16.767 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-11T21:44:16.767 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:44:16.767 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:44:16.767 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-03-11 21:44:07.088429694 +0000 2024-03-11T21:44:16.767 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-03-11 21:43:44.592932136 +0000 2024-03-11T21:44:16.767 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-03-11 21:43:44.592932136 +0000 2024-03-11T21:44:16.767 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-03-11 21:43:44.592932136 +0000 2024-03-11T21:44:16.767 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T21:44:16.836 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-03-11T21:44:16.836 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-03-11T21:44:16.836 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000103772 s, 4.9 MB/s 2024-03-11T21:44:16.838 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T21:44:16.898 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 2024-03-11T21:44:16.955 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T21:44:16.955 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:44:16.955 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-11T21:44:16.956 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:44:16.956 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:44:16.956 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-03-11 21:44:07.088429694 +0000 2024-03-11T21:44:16.956 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-03-11 21:43:44.932924542 +0000 2024-03-11T21:44:16.956 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-03-11 21:43:44.932924542 +0000 2024-03-11T21:44:16.956 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-03-11 21:43:44.932924542 +0000 2024-03-11T21:44:16.956 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T21:44:17.023 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-03-11T21:44:17.023 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-03-11T21:44:17.024 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.00011311 s, 4.5 MB/s 2024-03-11T21:44:17.025 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T21:44:17.083 DEBUG:teuthology.orchestra.run.smithi002:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T21:44:17.252 INFO:teuthology.orchestra.run.smithi002.stdout:loop 2024-03-11T21:44:17.253 INFO:tasks.nvme_loop:Connecting nvme_loop smithi002:/dev/vg_nvme/lv_1... 2024-03-11T21:44:17.254 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T21:44:17.293 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-03-11T21:44:17.331 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/vg_nvme/lv_11 2024-03-11T21:44:17.361 INFO:tasks.nvme_loop:Connecting nvme_loop smithi002:/dev/vg_nvme/lv_2... 2024-03-11T21:44:17.361 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T21:44:17.440 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-03-11T21:44:17.479 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/vg_nvme/lv_21 2024-03-11T21:44:17.520 INFO:tasks.nvme_loop:Connecting nvme_loop smithi002:/dev/vg_nvme/lv_3... 2024-03-11T21:44:17.520 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T21:44:17.598 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-03-11T21:44:17.636 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/vg_nvme/lv_31 2024-03-11T21:44:17.666 INFO:tasks.nvme_loop:Connecting nvme_loop smithi002:/dev/vg_nvme/lv_4... 2024-03-11T21:44:17.666 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T21:44:17.745 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-03-11T21:44:17.785 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/vg_nvme/lv_41 2024-03-11T21:44:17.822 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T21:44:17.822 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:44:17.878 DEBUG:teuthology.orchestra.run.smithi002:> sudo nvme list 2024-03-11T21:44:17.944 INFO:teuthology.orchestra.run.smithi002.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T21:44:17.945 INFO:teuthology.orchestra.run.smithi002.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T21:44:17.945 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/nvme0n1 PHMB751300EV480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-03-11T21:44:17.945 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/nvme1n1 f7f8e8ac88e01da4278c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:44:17.945 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/nvme2n1 71f6e8e759e933b228ad Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:44:17.945 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/nvme3n1 b530331a0339789641d7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:44:17.945 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/nvme4n1 13ee583bb841c752b78a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:44:17.946 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:44:17.946 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T21:44:17.946 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/scratch_devs 2024-03-11T21:44:18.013 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-11T21:44:18.013 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:44:18.032 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T21:44:18.032 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2024-03-11T21:44:18.089 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T21:44:18.089 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:44:18.089 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-11T21:44:18.090 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:44:18.090 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:44:18.090 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-11 21:44:07.118113798 +0000 2024-03-11T21:44:18.090 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-11 21:43:43.521682657 +0000 2024-03-11T21:44:18.090 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-11 21:43:43.521682657 +0000 2024-03-11T21:44:18.090 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-11 21:43:43.521682657 +0000 2024-03-11T21:44:18.091 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T21:44:18.158 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-11T21:44:18.158 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-11T21:44:18.158 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000214563 s, 2.4 MB/s 2024-03-11T21:44:18.159 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T21:44:18.218 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2024-03-11T21:44:18.274 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T21:44:18.275 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:44:18.275 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-11T21:44:18.275 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:44:18.275 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:44:18.275 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-11 21:44:07.118113798 +0000 2024-03-11T21:44:18.276 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-11 21:43:43.863674412 +0000 2024-03-11T21:44:18.276 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-11 21:43:43.863674412 +0000 2024-03-11T21:44:18.276 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-11 21:43:43.863674412 +0000 2024-03-11T21:44:18.276 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T21:44:18.342 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-11T21:44:18.342 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-11T21:44:18.343 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000212042 s, 2.4 MB/s 2024-03-11T21:44:18.343 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T21:44:18.403 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2024-03-11T21:44:18.458 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T21:44:18.459 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:44:18.459 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-11T21:44:18.459 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:44:18.459 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:44:18.459 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-11 21:44:07.118113798 +0000 2024-03-11T21:44:18.459 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-11 21:43:44.186666626 +0000 2024-03-11T21:44:18.459 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-11 21:43:44.186666626 +0000 2024-03-11T21:44:18.459 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-11 21:43:44.186666626 +0000 2024-03-11T21:44:18.459 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T21:44:18.525 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-11T21:44:18.525 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-11T21:44:18.525 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000191444 s, 2.7 MB/s 2024-03-11T21:44:18.526 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T21:44:18.581 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2024-03-11T21:44:18.636 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T21:44:18.637 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:44:18.637 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-11T21:44:18.637 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:44:18.637 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:44:18.637 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-11 21:44:07.118113798 +0000 2024-03-11T21:44:18.637 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-11 21:43:44.525658453 +0000 2024-03-11T21:44:18.638 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-11 21:43:44.525658453 +0000 2024-03-11T21:44:18.638 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-11 21:43:44.525658453 +0000 2024-03-11T21:44:18.638 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T21:44:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-11T21:44:18.702 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-11T21:44:18.703 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000181324 s, 2.8 MB/s 2024-03-11T21:44:18.704 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T21:44:18.759 DEBUG:teuthology.orchestra.run.smithi122:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T21:44:18.919 INFO:teuthology.orchestra.run.smithi122.stdout:loop 2024-03-11T21:44:18.920 INFO:tasks.nvme_loop:Connecting nvme_loop smithi122:/dev/vg_nvme/lv_1... 2024-03-11T21:44:18.920 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T21:44:18.958 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-11T21:44:18.995 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/vg_nvme/lv_11 2024-03-11T21:44:19.024 INFO:tasks.nvme_loop:Connecting nvme_loop smithi122:/dev/vg_nvme/lv_2... 2024-03-11T21:44:19.024 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T21:44:19.104 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-11T21:44:19.142 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/vg_nvme/lv_21 2024-03-11T21:44:19.172 INFO:tasks.nvme_loop:Connecting nvme_loop smithi122:/dev/vg_nvme/lv_3... 2024-03-11T21:44:19.172 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T21:44:19.251 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-11T21:44:19.290 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/vg_nvme/lv_31 2024-03-11T21:44:19.319 INFO:tasks.nvme_loop:Connecting nvme_loop smithi122:/dev/vg_nvme/lv_4... 2024-03-11T21:44:19.319 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T21:44:19.398 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-03-11T21:44:19.437 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/vg_nvme/lv_41 2024-03-11T21:44:19.467 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-11T21:44:19.467 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:44:19.523 DEBUG:teuthology.orchestra.run.smithi122:> sudo nvme list 2024-03-11T21:44:19.589 INFO:teuthology.orchestra.run.smithi122.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T21:44:19.589 INFO:teuthology.orchestra.run.smithi122.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T21:44:19.589 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/nvme0n1 PHFT6204010U400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101F0 2024-03-11T21:44:19.590 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/nvme1n1 712b7a166060c9b8f5c9 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:44:19.590 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/nvme2n1 efbeb717c8f073555ee3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:44:19.590 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/nvme3n1 738fefb58a792da8d2fc Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:44:19.590 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/nvme4n1 3a03a60bebc646033a82 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:44:19.591 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:44:19.591 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-11T21:44:19.591 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/scratch_devs 2024-03-11T21:44:19.658 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T21:44:19.658 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:44:19.677 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T21:44:19.677 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 2024-03-11T21:44:19.735 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T21:44:19.735 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:44:19.735 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T21:44:19.735 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:44:19.735 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:44:19.736 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-11 21:44:07.077691560 +0000 2024-03-11T21:44:19.736 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-11 21:43:43.525050021 +0000 2024-03-11T21:44:19.736 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-11 21:43:43.525050021 +0000 2024-03-11T21:44:19.736 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-11 21:43:43.525050021 +0000 2024-03-11T21:44:19.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T21:44:19.806 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-11T21:44:19.806 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-11T21:44:19.806 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.00022094 s, 2.3 MB/s 2024-03-11T21:44:19.808 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T21:44:19.866 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 2024-03-11T21:44:19.923 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T21:44:19.923 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:44:19.923 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-11T21:44:19.923 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:44:19.923 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:44:19.923 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-11 21:44:07.077691560 +0000 2024-03-11T21:44:19.923 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-11 21:43:43.862044892 +0000 2024-03-11T21:44:19.923 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-11 21:43:43.862044892 +0000 2024-03-11T21:44:19.923 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-11 21:43:43.862044892 +0000 2024-03-11T21:44:19.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T21:44:19.992 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-11T21:44:19.992 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-11T21:44:19.992 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000190723 s, 2.7 MB/s 2024-03-11T21:44:19.993 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T21:44:20.052 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 2024-03-11T21:44:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T21:44:20.109 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:44:20.109 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-11T21:44:20.109 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:44:20.110 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:44:20.110 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-11 21:44:07.077691560 +0000 2024-03-11T21:44:20.110 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-11 21:43:44.188039930 +0000 2024-03-11T21:44:20.110 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-11 21:43:44.188039930 +0000 2024-03-11T21:44:20.110 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-11 21:43:44.188039930 +0000 2024-03-11T21:44:20.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T21:44:20.177 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-11T21:44:20.178 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-11T21:44:20.178 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.00020061 s, 2.6 MB/s 2024-03-11T21:44:20.179 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T21:44:20.236 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 2024-03-11T21:44:20.293 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T21:44:20.293 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:44:20.293 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-11T21:44:20.293 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:44:20.293 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:44:20.293 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-11 21:44:07.077691560 +0000 2024-03-11T21:44:20.293 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-11 21:43:44.538034604 +0000 2024-03-11T21:44:20.293 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-11 21:43:44.538034604 +0000 2024-03-11T21:44:20.294 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-11 21:43:44.538034604 +0000 2024-03-11T21:44:20.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T21:44:20.359 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-11T21:44:20.359 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-11T21:44:20.359 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000217894 s, 2.3 MB/s 2024-03-11T21:44:20.361 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T21:44:20.417 DEBUG:teuthology.orchestra.run.smithi165:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T21:44:20.602 INFO:teuthology.orchestra.run.smithi165.stdout:loop 2024-03-11T21:44:20.603 INFO:tasks.nvme_loop:Connecting nvme_loop smithi165:/dev/vg_nvme/lv_1... 2024-03-11T21:44:20.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T21:44:20.641 INFO:teuthology.orchestra.run.smithi165.stdout:1 2024-03-11T21:44:20.678 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/vg_nvme/lv_11 2024-03-11T21:44:20.715 INFO:tasks.nvme_loop:Connecting nvme_loop smithi165:/dev/vg_nvme/lv_2... 2024-03-11T21:44:20.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T21:44:20.794 INFO:teuthology.orchestra.run.smithi165.stdout:1 2024-03-11T21:44:20.833 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/vg_nvme/lv_21 2024-03-11T21:44:20.872 INFO:tasks.nvme_loop:Connecting nvme_loop smithi165:/dev/vg_nvme/lv_3... 2024-03-11T21:44:20.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T21:44:20.951 INFO:teuthology.orchestra.run.smithi165.stdout:1 2024-03-11T21:44:20.990 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/vg_nvme/lv_31 2024-03-11T21:44:21.028 INFO:tasks.nvme_loop:Connecting nvme_loop smithi165:/dev/vg_nvme/lv_4... 2024-03-11T21:44:21.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T21:44:21.109 INFO:teuthology.orchestra.run.smithi165.stdout:1 2024-03-11T21:44:21.148 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/vg_nvme/lv_41 2024-03-11T21:44:21.196 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T21:44:21.196 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:44:21.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo nvme list 2024-03-11T21:44:21.317 INFO:teuthology.orchestra.run.smithi165.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T21:44:21.317 INFO:teuthology.orchestra.run.smithi165.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T21:44:21.317 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/nvme0n1 CVFT623300CE400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-11T21:44:21.317 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/nvme1n1 0b67468dd008f9ae3921 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:44:21.317 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/nvme2n1 2f621fa837f0ee352d1d Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:44:21.317 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/nvme3n1 c2cf2f321488040ebe03 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:44:21.317 INFO:teuthology.orchestra.run.smithi165.stdout:/dev/nvme4n1 517b49c11fe5ca538c85 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T21:44:21.318 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:44:21.318 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T21:44:21.318 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/scratch_devs 2024-03-11T21:44:21.385 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T21:44:21.487 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'global': {'mon election default strategy': 3}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-11T21:44:21.488 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:44:21.488 INFO:tasks.cephadm:Cluster fsid is 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:44:21.488 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T21:44:21.488 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.2', 'mon.b': '172.21.15.122', 'mon.c': '172.21.15.165'} 2024-03-11T21:44:21.488 INFO:tasks.cephadm:First mon is mon.a on smithi002 2024-03-11T21:44:21.488 INFO:tasks.cephadm:First mgr is y 2024-03-11T21:44:21.488 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T21:44:21.489 DEBUG:teuthology.orchestra.run.smithi002:> sudo hostname $(hostname -s) 2024-03-11T21:44:21.521 DEBUG:teuthology.orchestra.run.smithi122:> sudo hostname $(hostname -s) 2024-03-11T21:44:21.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo hostname $(hostname -s) 2024-03-11T21:44:21.581 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-11T21:44:21.581 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:44:21.728 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77431/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:08:19.449750', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77429/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:06:59.696700', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-11T21:44:21.943 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 94a539707733325322964c6a187c6a1b41be1daf from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:44:21.945 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm 2024-03-11T21:44:21.945 DEBUG:teuthology.orchestra.run.smithi002:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T21:44:22.587 INFO:teuthology.orchestra.run.smithi002.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 21:44 /home/ubuntu/cephtest/cephadm 2024-03-11T21:44:22.587 DEBUG:teuthology.orchestra.run.smithi122:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T21:44:23.354 INFO:teuthology.orchestra.run.smithi122.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 21:44 /home/ubuntu/cephtest/cephadm 2024-03-11T21:44:23.354 DEBUG:teuthology.orchestra.run.smithi165:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T21:44:23.868 INFO:teuthology.orchestra.run.smithi165.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 21:44 /home/ubuntu/cephtest/cephadm 2024-03-11T21:44:23.869 DEBUG:teuthology.orchestra.run.smithi002:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T21:44:23.887 DEBUG:teuthology.orchestra.run.smithi122:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T21:44:23.906 DEBUG:teuthology.orchestra.run.smithi165:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T21:44:23.939 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf on all hosts... 2024-03-11T21:44:23.939 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T21:44:23.942 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T21:44:23.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T21:44:24.754 INFO:teuthology.orchestra.run.smithi002.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T21:44:24.779 INFO:teuthology.orchestra.run.smithi165.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T21:44:24.792 INFO:teuthology.orchestra.run.smithi122.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T21:44:52.362 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-11T21:44:52.362 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T21:44:52.362 INFO:teuthology.orchestra.run.smithi122.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T21:44:52.362 INFO:teuthology.orchestra.run.smithi122.stdout: "repo_digests": [ 2024-03-11T21:44:52.362 INFO:teuthology.orchestra.run.smithi122.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T21:44:52.363 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-11T21:44:52.363 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-11T21:45:22.934 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-03-11T21:45:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T21:45:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T21:45:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: "repo_digests": [ 2024-03-11T21:45:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T21:45:22.935 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-03-11T21:45:22.935 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-03-11T21:45:32.569 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-03-11T21:45:32.569 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T21:45:32.569 INFO:teuthology.orchestra.run.smithi002.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T21:45:32.569 INFO:teuthology.orchestra.run.smithi002.stdout: "repo_digests": [ 2024-03-11T21:45:32.569 INFO:teuthology.orchestra.run.smithi002.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T21:45:32.569 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-03-11T21:45:32.569 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-03-11T21:45:32.593 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph 2024-03-11T21:45:32.625 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph 2024-03-11T21:45:32.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph 2024-03-11T21:45:32.698 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /etc/ceph 2024-03-11T21:45:32.723 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /etc/ceph 2024-03-11T21:45:32.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /etc/ceph 2024-03-11T21:45:32.785 INFO:tasks.cephadm:Writing seed config... 2024-03-11T21:45:32.786 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T21:45:32.787 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T21:45:32.787 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-03-11T21:45:32.787 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-11T21:45:32.787 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T21:45:32.787 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T21:45:32.787 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T21:45:32.787 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T21:45:32.787 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T21:45:32.787 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T21:45:32.788 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T21:45:32.788 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T21:45:32.802 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 85326acc-dff0-11ee-95c8-87774f69a715 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-11T21:45:32.802 DEBUG:teuthology.orchestra.run.smithi002:mon.a> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T21:45:32.844 DEBUG:teuthology.orchestra.run.smithi002:mgr.y> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T21:45:32.886 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T21:45:32.887 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf -v bootstrap --fsid 85326acc-dff0-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.2 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:45:33.082 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:45:33.082 INFO:teuthology.orchestra.run.smithi002.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf', '-v', 'bootstrap', '--fsid', '85326acc-dff0-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.2', '--skip-admin-label'] 2024-03-11T21:45:33.107 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T21:45:33.107 INFO:teuthology.orchestra.run.smithi002.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T21:45:33.107 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying podman|docker is present... 2024-03-11T21:45:33.130 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T21:45:33.130 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying lvm2 is present... 2024-03-11T21:45:33.130 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying time synchronization is in place... 2024-03-11T21:45:33.137 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T21:45:33.137 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T21:45:33.143 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T21:45:33.144 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-03-11T21:45:33.149 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout enabled 2024-03-11T21:45:33.155 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout active 2024-03-11T21:45:33.155 INFO:teuthology.orchestra.run.smithi002.stdout:Unit chronyd.service is enabled and running 2024-03-11T21:45:33.155 INFO:teuthology.orchestra.run.smithi002.stdout:Repeating the final host check... 2024-03-11T21:45:33.178 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T21:45:33.178 INFO:teuthology.orchestra.run.smithi002.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-11T21:45:33.178 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl is present 2024-03-11T21:45:33.178 INFO:teuthology.orchestra.run.smithi002.stdout:lvcreate is present 2024-03-11T21:45:33.184 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T21:45:33.184 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T21:45:33.190 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T21:45:33.190 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-03-11T21:45:33.196 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout enabled 2024-03-11T21:45:33.202 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout active 2024-03-11T21:45:33.202 INFO:teuthology.orchestra.run.smithi002.stdout:Unit chronyd.service is enabled and running 2024-03-11T21:45:33.202 INFO:teuthology.orchestra.run.smithi002.stdout:Host looks OK 2024-03-11T21:45:33.202 INFO:teuthology.orchestra.run.smithi002.stdout:Cluster fsid: 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:45:33.202 INFO:teuthology.orchestra.run.smithi002.stdout:Acquiring lock 139963651279456 on /run/cephadm/85326acc-dff0-11ee-95c8-87774f69a715.lock 2024-03-11T21:45:33.202 INFO:teuthology.orchestra.run.smithi002.stdout:Lock 139963651279456 acquired on /run/cephadm/85326acc-dff0-11ee-95c8-87774f69a715.lock 2024-03-11T21:45:33.203 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying IP 172.21.15.2 port 3300 ... 2024-03-11T21:45:33.203 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying IP 172.21.15.2 port 6789 ... 2024-03-11T21:45:33.203 INFO:teuthology.orchestra.run.smithi002.stdout:Base mon IP(s) is [172.21.15.2:3300, 172.21.15.2:6789], mon addrv is [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2024-03-11T21:45:33.206 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.2 metric 100 2024-03-11T21:45:33.206 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.2 metric 100 2024-03-11T21:45:33.208 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T21:45:33.208 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-11T21:45:33.209 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-11T21:45:33.211 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T21:45:33.211 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T21:45:33.211 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T21:45:33.211 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-11T21:45:33.211 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-03-11T21:45:33.211 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T21:45:33.212 INFO:teuthology.orchestra.run.smithi002.stdout:Mon IP `172.21.15.2` is in CIDR network `172.21.0.0/20` 2024-03-11T21:45:33.212 INFO:teuthology.orchestra.run.smithi002.stdout:Mon IP `172.21.15.2` is in CIDR network `172.21.0.0/20` 2024-03-11T21:45:33.212 INFO:teuthology.orchestra.run.smithi002.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-11T21:45:33.212 INFO:teuthology.orchestra.run.smithi002.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T21:45:33.213 INFO:teuthology.orchestra.run.smithi002.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T21:45:33.671 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T21:45:33.671 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T21:45:33.671 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Getting image source signatures 2024-03-11T21:45:33.671 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Copying blob sha256:0b3c7bee4e66dab4b5376aaf0baf8ba5fc1c3e66c27dc601157782d617bd9dc6 2024-03-11T21:45:33.671 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-11T21:45:33.671 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Copying config sha256:a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T21:45:33.671 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-11T21:45:34.508 INFO:teuthology.orchestra.run.smithi002.stdout:ceph: stdout ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T21:45:34.508 INFO:teuthology.orchestra.run.smithi002.stdout:Ceph version: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T21:45:34.508 INFO:teuthology.orchestra.run.smithi002.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T21:45:35.564 INFO:teuthology.orchestra.run.smithi002.stdout:stat: stdout 167 167 2024-03-11T21:45:35.564 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial keys... 2024-03-11T21:45:36.703 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph-authtool: stdout AQCAe+9lRRIwABAAa8uF5FSt3dc5drPy5Plqsw== 2024-03-11T21:45:37.633 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph-authtool: stdout AQCBe+9lXzT0BBAAM3LBrVDi9EPeI2Ou/EVGwA== 2024-03-11T21:45:38.622 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph-authtool: stdout AQCCe+9l9b9QARAAmAXVWs2KwZ9aX3xthMNJ9g== 2024-03-11T21:45:38.623 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial monmap... 2024-03-11T21:45:39.461 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T21:45:39.461 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-11T21:45:39.461 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:45:39.461 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T21:45:39.461 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool for a [v2:172.21.15.2:3300,v1:172.21.15.2:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T21:45:39.461 INFO:teuthology.orchestra.run.smithi002.stdout:setting min_mon_release = quincy 2024-03-11T21:45:39.461 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: set fsid to 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:45:39.461 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T21:45:39.461 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:45:39.462 INFO:teuthology.orchestra.run.smithi002.stdout:Creating mon... 2024-03-11T21:45:40.399 INFO:teuthology.orchestra.run.smithi002.stdout:create mon.a on 2024-03-11T21:45:40.872 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T21:45:41.047 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-85326acc-dff0-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-85326acc-dff0-11ee-95c8-87774f69a715.target. 2024-03-11T21:45:41.047 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-85326acc-dff0-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-85326acc-dff0-11ee-95c8-87774f69a715.target. 2024-03-11T21:45:41.273 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-85326acc-dff0-11ee-95c8-87774f69a715@mon.a 2024-03-11T21:45:41.273 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to reset failed state of unit ceph-85326acc-dff0-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-85326acc-dff0-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T21:45:41.433 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-85326acc-dff0-11ee-95c8-87774f69a715.target.wants/ceph-85326acc-dff0-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-85326acc-dff0-11ee-95c8-87774f69a715@.service. 2024-03-11T21:45:41.690 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:41 smithi002 podman[25236]: 2024-03-11 21:45:41.587531882 +0000 UTC m=+0.017554754 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:45:41.690 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:41 smithi002 podman[25236]: 2024-03-11 21:45:41.700962385 +0000 UTC m=+0.130985296 container create 39c842158258b1bd9ea869fca03ce2d9b2966d366e0cc96c6454b9d36ec1b1da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-94a5397) 2024-03-11T21:45:41.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:41 smithi002 podman[25236]: 2024-03-11 21:45:41.920071673 +0000 UTC m=+0.350094545 container init 39c842158258b1bd9ea869fca03ce2d9b2966d366e0cc96c6454b9d36ec1b1da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-11T21:45:41.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:41 smithi002 podman[25236]: 2024-03-11 21:45:41.925607037 +0000 UTC m=+0.355629909 container start 39c842158258b1bd9ea869fca03ce2d9b2966d366e0cc96c6454b9d36ec1b1da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS) 2024-03-11T21:45:41.998 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:45:41.998 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2024-03-11T21:45:42.004 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:45:42.004 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-03-11T21:45:42.004 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2024-03-11T21:45:42.004 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T21:45:42.004 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mon to start... 2024-03-11T21:45:42.004 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mon... 2024-03-11T21:45:42.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:41 smithi002 ceph-mon[25274]: mkfs 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:45:42.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:41 smithi002 bash[25236]: 39c842158258b1bd9ea869fca03ce2d9b2966d366e0cc96c6454b9d36ec1b1da 2024-03-11T21:45:42.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:41 smithi002 ceph-mon[25274]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T21:45:42.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:41 smithi002 systemd[1]: Started Ceph mon.a for 85326acc-dff0-11ee-95c8-87774f69a715. 2024-03-11T21:45:43.207 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:42 smithi002 ceph-mon[25274]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T21:45:43.207 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:42 smithi002 ceph-mon[25274]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:45:43.207 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:42 smithi002 ceph-mon[25274]: fsmap 2024-03-11T21:45:43.208 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:42 smithi002 ceph-mon[25274]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T21:45:43.208 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:42 smithi002 ceph-mon[25274]: mgrmap e1: no daemons active 2024-03-11T21:45:43.208 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:42 smithi002 ceph-mon[25274]: from='client.? 172.21.15.2:0/4231676540' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-11T21:45:43.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T21:45:43.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout id: 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:45:43.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T21:45:43.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T21:45:43.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout services: 2024-03-11T21:45:43.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.659178s) 2024-03-11T21:45:43.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T21:45:43.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T21:45:43.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T21:45:43.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout data: 2024-03-11T21:45:43.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T21:45:43.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T21:45:43.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T21:45:43.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T21:45:43.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T21:45:43.215 INFO:teuthology.orchestra.run.smithi002.stdout:mon is available 2024-03-11T21:45:43.215 INFO:teuthology.orchestra.run.smithi002.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T21:45:44.071 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:43 smithi002 ceph-mon[25274]: from='client.? 172.21.15.2:0/1740914198' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T21:45:44.071 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:43 smithi002 ceph-mon[25274]: from='client.? 172.21.15.2:0/1740914198' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T21:45:44.251 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T21:45:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T21:45:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout fsid = 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:45:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2024-03-11T21:45:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T21:45:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T21:45:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T21:45:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T21:45:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T21:45:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T21:45:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-03-11T21:45:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T21:45:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T21:45:44.252 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T21:45:44.253 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T21:45:44.253 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T21:45:44.253 INFO:teuthology.orchestra.run.smithi002.stdout:Generating new minimal ceph.conf... 2024-03-11T21:45:45.393 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:44 smithi002 ceph-mon[25274]: from='client.? 172.21.15.2:0/4001640533' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:45:45.406 INFO:teuthology.orchestra.run.smithi002.stdout:Restarting the monitor... 2024-03-11T21:45:45.667 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:45 smithi002 systemd[1]: Stopping Ceph mon.a for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:45:46.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:45 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a[25251]: 2024-03-11T21:45:45.676+0000 7f7fe7ddc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:45:46.006 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:45 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a[25251]: 2024-03-11T21:45:45.676+0000 7f7fe7ddc700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T21:45:46.006 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:45 smithi002 podman[25609]: 2024-03-11 21:45:45.769082086 +0000 UTC m=+0.186729665 container died 39c842158258b1bd9ea869fca03ce2d9b2966d366e0cc96c6454b9d36ec1b1da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-11T21:45:46.272 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 podman[25609]: 2024-03-11 21:45:46.098521608 +0000 UTC m=+0.516169195 container cleanup 39c842158258b1bd9ea869fca03ce2d9b2966d366e0cc96c6454b9d36ec1b1da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-94a5397) 2024-03-11T21:45:46.272 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 bash[25609]: ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a 2024-03-11T21:45:46.522 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 podman[25628]: 2024-03-11 21:45:46.282823946 +0000 UTC m=+0.513363241 container remove 39c842158258b1bd9ea869fca03ce2d9b2966d366e0cc96c6454b9d36ec1b1da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-11T21:45:46.523 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 systemd[1]: ceph-85326acc-dff0-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T21:45:46.523 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 systemd[1]: Stopped Ceph mon.a for 85326acc-dff0-11ee-95c8-87774f69a715. 2024-03-11T21:45:46.523 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 systemd[1]: Starting Ceph mon.a for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:45:46.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 podman[25727]: 2024-03-11 21:45:46.7422227 +0000 UTC m=+0.113067855 container create 64d5c323ec3ba1e2601d372ddbbdd17ae8dda1095df3a2e2cc52056fc0ace318 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:45:46.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 podman[25727]: 2024-03-11 21:45:46.647471883 +0000 UTC m=+0.018317006 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:45:47.004 INFO:teuthology.orchestra.run.smithi002.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-11T21:45:47.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 podman[25727]: 2024-03-11 21:45:46.935970845 +0000 UTC m=+0.306815961 container init 64d5c323ec3ba1e2601d372ddbbdd17ae8dda1095df3a2e2cc52056fc0ace318 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-11T21:45:47.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 podman[25727]: 2024-03-11 21:45:46.941001793 +0000 UTC m=+0.311846907 container start 64d5c323ec3ba1e2601d372ddbbdd17ae8dda1095df3a2e2cc52056fc0ace318 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-11T21:45:47.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T21:45:47.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 7 2024-03-11T21:45:47.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: pidfile_write: ignore empty --pid-file 2024-03-11T21:45:47.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: load: jerasure load: lrc 2024-03-11T21:45:47.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: RocksDB version: 7.9.2 2024-03-11T21:45:47.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Git sha 0 2024-03-11T21:45:47.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T21:45:47.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: DB SUMMARY 2024-03-11T21:45:47.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: DB Session ID: OQ154O9WH25586ORXXFV 2024-03-11T21:45:47.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: CURRENT file: CURRENT 2024-03-11T21:45:47.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T21:45:47.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-11T21:45:47.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T21:45:47.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84601 ; 2024-03-11T21:45:47.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.error_if_exists: 0 2024-03-11T21:45:47.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.create_if_missing: 0 2024-03-11T21:45:47.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.paranoid_checks: 1 2024-03-11T21:45:47.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T21:45:47.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T21:45:47.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T21:45:47.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.env: 0x5586e565bae0 2024-03-11T21:45:47.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T21:45:47.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.info_log: 0x5586e65578c0 2024-03-11T21:45:47.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T21:45:47.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.statistics: (nil) 2024-03-11T21:45:47.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.use_fsync: 0 2024-03-11T21:45:47.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_log_file_size: 0 2024-03-11T21:45:47.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T21:45:47.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T21:45:47.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T21:45:47.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T21:45:47.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.allow_fallocate: 1 2024-03-11T21:45:47.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T21:45:47.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T21:45:47.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.use_direct_reads: 0 2024-03-11T21:45:47.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T21:45:47.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T21:45:47.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.db_log_dir: 2024-03-11T21:45:47.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.wal_dir: 2024-03-11T21:45:47.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T21:45:47.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T21:45:47.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T21:45:47.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T21:45:47.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T21:45:47.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T21:45:47.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T21:45:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T21:45:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.write_buffer_manager: 0x5586e652b400 2024-03-11T21:45:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T21:45:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T21:45:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T21:45:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T21:45:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T21:45:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T21:45:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T21:45:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T21:45:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.unordered_write: 0 2024-03-11T21:45:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T21:45:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T21:45:47.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T21:45:47.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T21:45:47.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.row_cache: None 2024-03-11T21:45:47.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.wal_filter: None 2024-03-11T21:45:47.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T21:45:47.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T21:45:47.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.two_write_queues: 0 2024-03-11T21:45:47.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T21:45:47.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.wal_compression: 0 2024-03-11T21:45:47.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.atomic_flush: 0 2024-03-11T21:45:47.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T21:45:47.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T21:45:47.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T21:45:47.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.log_readahead_size: 0 2024-03-11T21:45:47.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T21:45:47.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T21:45:47.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T21:45:47.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T21:45:47.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T21:45:47.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T21:45:47.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T21:45:47.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_background_jobs: 2 2024-03-11T21:45:47.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_background_compactions: -1 2024-03-11T21:45:47.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_subcompactions: 1 2024-03-11T21:45:47.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T21:45:47.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T21:45:47.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T21:45:47.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T21:45:47.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T21:45:47.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T21:45:47.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T21:45:47.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T21:45:47.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_open_files: -1 2024-03-11T21:45:47.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T21:45:47.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T21:45:47.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T21:45:47.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T21:45:47.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_background_flushes: -1 2024-03-11T21:45:47.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Compression algorithms supported: 2024-03-11T21:45:47.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T21:45:47.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: kZSTD supported: 0 2024-03-11T21:45:47.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: kXpressCompression supported: 0 2024-03-11T21:45:47.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T21:45:47.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: kZlibCompression supported: 1 2024-03-11T21:45:47.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: kSnappyCompression supported: 1 2024-03-11T21:45:47.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: kLZ4Compression supported: 1 2024-03-11T21:45:47.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: kBZip2Compression supported: 0 2024-03-11T21:45:47.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T21:45:47.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T21:45:47.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-11T21:45:47.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T21:45:47.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T21:45:47.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.merge_operator: 2024-03-11T21:45:47.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compaction_filter: None 2024-03-11T21:45:47.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T21:45:47.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T21:45:47.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T21:45:47.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T21:45:47.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5586e65579c0) 2024-03-11T21:45:47.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks: 1 2024-03-11T21:45:47.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T21:45:47.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T21:45:47.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_top_level_index_and_filter: 1 2024-03-11T21:45:47.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_type: 0 2024-03-11T21:45:47.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_index_type: 0 2024-03-11T21:45:47.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_shortening: 1 2024-03-11T21:45:47.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T21:45:47.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: checksum: 4 2024-03-11T21:45:47.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: no_block_cache: 0 2024-03-11T21:45:47.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache: 0x5586e655b090 2024-03-11T21:45:47.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_name: BinnedLRUCache 2024-03-11T21:45:47.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_options: 2024-03-11T21:45:47.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: capacity : 536870912 2024-03-11T21:45:47.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_shard_bits : 4 2024-03-11T21:45:47.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: strict_capacity_limit : 0 2024-03-11T21:45:47.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: high_pri_pool_ratio: 0.000 2024-03-11T21:45:47.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_compressed: (nil) 2024-03-11T21:45:47.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: persistent_cache: (nil) 2024-03-11T21:45:47.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size: 4096 2024-03-11T21:45:47.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size_deviation: 10 2024-03-11T21:45:47.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_restart_interval: 16 2024-03-11T21:45:47.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_block_restart_interval: 1 2024-03-11T21:45:47.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: metadata_block_size: 4096 2024-03-11T21:45:47.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: partition_filters: 0 2024-03-11T21:45:47.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: use_delta_encoding: 1 2024-03-11T21:45:47.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: filter_policy: bloomfilter 2024-03-11T21:45:47.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: whole_key_filtering: 1 2024-03-11T21:45:47.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: verify_compression: 0 2024-03-11T21:45:47.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: read_amp_bytes_per_bit: 0 2024-03-11T21:45:47.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: format_version: 5 2024-03-11T21:45:47.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_index_compression: 1 2024-03-11T21:45:47.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_align: 0 2024-03-11T21:45:47.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: max_auto_readahead_size: 262144 2024-03-11T21:45:47.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: prepopulate_block_cache: 0 2024-03-11T21:45:47.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: initial_auto_readahead_size: 8192 2024-03-11T21:45:47.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T21:45:47.237 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T21:45:47.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T21:45:47.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compression: NoCompression 2024-03-11T21:45:47.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T21:45:47.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T21:45:47.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T21:45:47.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.num_levels: 7 2024-03-11T21:45:47.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T21:45:47.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T21:45:47.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T21:45:47.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T21:45:47.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T21:45:47.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T21:45:47.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T21:45:47.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:45:47.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T21:45:47.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T21:45:47.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:45:47.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:45:47.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T21:45:47.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T21:45:47.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T21:45:47.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T21:45:47.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:45:47.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:45:47.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T21:45:47.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T21:45:47.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:45:47.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T21:45:47.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T21:45:47.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T21:45:47.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T21:45:47.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T21:45:47.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T21:45:47.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T21:45:47.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T21:45:47.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T21:45:47.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T21:45:47.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T21:45:47.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T21:45:47.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T21:45:47.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T21:45:47.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T21:45:47.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T21:45:47.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T21:45:47.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T21:45:47.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T21:45:47.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T21:45:47.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T21:45:47.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T21:45:47.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T21:45:47.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T21:45:47.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T21:45:47.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T21:45:47.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T21:45:47.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T21:45:47.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T21:45:47.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T21:45:47.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T21:45:47.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T21:45:47.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.table_properties_collectors: 2024-03-11T21:45:47.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.inplace_update_support: 0 2024-03-11T21:45:47.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T21:45:47.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T21:45:47.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T21:45:47.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T21:45:47.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.bloom_locality: 0 2024-03-11T21:45:47.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.max_successive_merges: 0 2024-03-11T21:45:47.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T21:45:47.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T21:45:47.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T21:45:47.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T21:45:47.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.ttl: 2592000 2024-03-11T21:45:47.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T21:45:47.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T21:45:47.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T21:45:47.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.enable_blob_files: false 2024-03-11T21:45:47.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.min_blob_size: 0 2024-03-11T21:45:47.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T21:45:47.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T21:45:47.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T21:45:47.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T21:45:47.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T21:45:47.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T21:45:47.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T21:45:47.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T21:45:47.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-11T21:45:47.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-11T21:45:47.244 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6088a45e-43d3-4e32-87e4-b2e6efc621ad 2024-03-11T21:45:47.244 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710193546970058, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-11T21:45:47.244 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-11T21:45:47.244 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710193546970888, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81226, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79413, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73722, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710193546, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6088a45e-43d3-4e32-87e4-b2e6efc621ad", "db_session_id": "OQ154O9WH25586ORXXFV", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-11T21:45:47.244 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710193546970973, "job": 1, "event": "recovery_finished"} 2024-03-11T21:45:47.244 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-11T21:45:47.244 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T21:45:47.244 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5586e664c000 2024-03-11T21:45:47.244 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: DB pointer 0x5586e6636000 2024-03-11T21:45:47.244 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T21:45:47.244 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T21:45:47.244 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** DB Stats ** 2024-03-11T21:45:47.244 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:45:47.244 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T21:45:47.245 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:45:47.245 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:45:47.245 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T21:45:47.245 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:45:47.245 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:45:47.245 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-03-11T21:45:47.245 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-03-11T21:45:47.245 INFO:journalctl@ceph.mon.a.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:45:47.245 INFO:journalctl@ceph.mon.a.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T21:45:47.245 INFO:journalctl@ceph.mon.a.smithi002.stdout: L0 2/0 81.16 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:45:47.245 INFO:journalctl@ceph.mon.a.smithi002.stdout: Sum 2/0 81.16 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:45:47.245 INFO:journalctl@ceph.mon.a.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:45:47.245 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-03-11T21:45:47.245 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-03-11T21:45:47.246 INFO:journalctl@ceph.mon.a.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:45:47.246 INFO:journalctl@ceph.mon.a.smithi002.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T21:45:47.246 INFO:journalctl@ceph.mon.a.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:45:47.246 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-03-11T21:45:47.246 INFO:journalctl@ceph.mon.a.smithi002.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T21:45:47.246 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-03-11T21:45:47.246 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:45:47.246 INFO:journalctl@ceph.mon.a.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T21:45:47.246 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T21:45:47.246 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T21:45:47.246 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T21:45:47.246 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T21:45:47.246 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative compaction: 0.00 GB write, 27.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:45:47.246 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval compaction: 0.00 GB write, 27.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:45:47.247 INFO:journalctl@ceph.mon.a.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T21:45:47.247 INFO:journalctl@ceph.mon.a.smithi002.stdout: Block cache BinnedLRUCache@0x5586e655b090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-11T21:45:47.247 INFO:journalctl@ceph.mon.a.smithi002.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-11T21:45:47.247 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-03-11T21:45:47.247 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T21:45:47.247 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: starting mon.a rank 0 at public addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] at bind addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:45:47.247 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: mon.a@-1(???) e1 preinit fsid 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:45:47.247 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: mon.a@-1(???).mds e1 new map 2024-03-11T21:45:47.247 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: mon.a@-1(???).mds e1 print_map 2024-03-11T21:45:47.247 INFO:journalctl@ceph.mon.a.smithi002.stdout: e1 2024-03-11T21:45:47.247 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T21:45:47.247 INFO:journalctl@ceph.mon.a.smithi002.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T21:45:47.247 INFO:journalctl@ceph.mon.a.smithi002.stdout: legacy client fscid: -1 2024-03-11T21:45:47.248 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-03-11T21:45:47.248 INFO:journalctl@ceph.mon.a.smithi002.stdout: No filesystems configured 2024-03-11T21:45:47.248 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T21:45:47.248 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:45:47.248 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:45:47.248 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:45:47.248 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:46 smithi002 ceph-mon[25764]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T21:45:47.248 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:47 smithi002 bash[25727]: 64d5c323ec3ba1e2601d372ddbbdd17ae8dda1095df3a2e2cc52056fc0ace318 2024-03-11T21:45:47.248 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:47 smithi002 systemd[1]: Started Ceph mon.a for 85326acc-dff0-11ee-95c8-87774f69a715. 2024-03-11T21:45:48.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:47 smithi002 ceph-mon[25764]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T21:45:48.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:47 smithi002 ceph-mon[25764]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:45:48.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:47 smithi002 ceph-mon[25764]: fsmap 2024-03-11T21:45:48.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:47 smithi002 ceph-mon[25764]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T21:45:48.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:47 smithi002 ceph-mon[25764]: mgrmap e1: no daemons active 2024-03-11T21:45:48.106 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:47 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/2732908488' entity='client.admin' 2024-03-11T21:45:48.108 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T21:45:48.130 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:45:48.131 INFO:teuthology.orchestra.run.smithi002.stdout:Creating mgr... 2024-03-11T21:45:48.131 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-11T21:45:48.131 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-11T21:45:48.386 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-85326acc-dff0-11ee-95c8-87774f69a715@mgr.y 2024-03-11T21:45:48.386 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to reset failed state of unit ceph-85326acc-dff0-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-85326acc-dff0-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-11T21:45:48.535 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-85326acc-dff0-11ee-95c8-87774f69a715.target.wants/ceph-85326acc-dff0-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-85326acc-dff0-11ee-95c8-87774f69a715@.service. 2024-03-11T21:45:48.643 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:48 smithi002 systemd[1]: Starting Ceph mgr.y for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:45:49.026 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:48 smithi002 podman[26009]: 2024-03-11 21:45:48.776863709 +0000 UTC m=+0.110653764 container create 0aba717a9141167a8cc57fc640ddbb6abca20f823152e3eb099e9acced4aa2d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-11T21:45:49.027 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:48 smithi002 podman[26009]: 2024-03-11 21:45:48.682826296 +0000 UTC m=+0.016616357 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:45:49.027 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:48 smithi002 podman[26009]: 2024-03-11 21:45:48.970416275 +0000 UTC m=+0.304206333 container init 0aba717a9141167a8cc57fc640ddbb6abca20f823152e3eb099e9acced4aa2d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:45:49.027 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:48 smithi002 podman[26009]: 2024-03-11 21:45:48.975480484 +0000 UTC m=+0.309270540 container start 0aba717a9141167a8cc57fc640ddbb6abca20f823152e3eb099e9acced4aa2d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5) 2024-03-11T21:45:49.027 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:49 smithi002 bash[26009]: 0aba717a9141167a8cc57fc640ddbb6abca20f823152e3eb099e9acced4aa2d8 2024-03-11T21:45:49.040 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:45:49.040 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2024-03-11T21:45:49.047 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:45:49.047 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-03-11T21:45:49.047 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2024-03-11T21:45:49.047 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T21:45:49.054 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:45:49.054 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2024-03-11T21:45:49.062 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:45:49.062 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-03-11T21:45:49.062 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2024-03-11T21:45:49.062 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-11T21:45:49.062 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr to start... 2024-03-11T21:45:49.062 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr... 2024-03-11T21:45:49.299 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:49 smithi002 systemd[1]: Started Ceph mgr.y for 85326acc-dff0-11ee-95c8-87774f69a715. 2024-03-11T21:45:49.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:49 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/2310708938' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:45:49.893 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:49 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:49.568+0000 7fa7532fc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:45:49.893 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:49 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:49.672+0000 7fa7532fc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:45:50.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T21:45:50.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:45:50.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "85326acc-dff0-11ee-95c8-87774f69a715", 2024-03-11T21:45:50.257 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:45:50.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:45:50.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:45:50.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:45:50.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:50.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:45:50.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:45:50.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:45:50.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:45:50.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:45:50.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:45:50.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:45:50.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-11T21:45:50.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:45:50.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:45:50.258 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:45:50.259 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:45:50.260 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:45:50.260 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:45:50.260 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:45:50.260 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:45:50.260 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:50.260 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:45:50.260 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:45:50.260 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:45:50.260 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:45:50.260 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:50.260 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:45:50.260 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:45:50.260 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:45:50.260 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:45:50.260 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:45:50.261 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:45:50.261 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:45:50.261 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:45:50.261 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:45:50.261 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:50.261 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:45:50.261 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:45:50.261 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:45:41.992595+0000", 2024-03-11T21:45:50.261 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:45:50.261 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:50.261 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:45:50.261 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:45:50.261 INFO:teuthology.orchestra.run.smithi002.stdout:mgr not available, waiting (1/15)... 2024-03-11T21:45:50.643 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:50 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:50.343+0000 7fa7532fc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:45:50.644 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:50 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:50.438+0000 7fa7532fc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:45:50.644 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:50 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:50.581+0000 7fa7532fc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:45:51.022 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:50 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:50.679+0000 7fa7532fc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:45:51.022 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:50 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:50.764+0000 7fa7532fc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:45:51.022 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:51 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:51.031+0000 7fa7532fc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:45:51.318 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:51 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:51.143+0000 7fa7532fc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:45:51.318 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:51 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:51.231+0000 7fa7532fc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:45:51.318 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:51 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:51.327+0000 7fa7532fc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:45:51.643 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:51 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:51.505+0000 7fa7532fc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:45:52.632 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:52 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:52.518+0000 7fa7532fc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:45:52.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:52 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/349288234' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:45:52.893 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:52 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:52.653+0000 7fa7532fc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:45:53.283 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:53 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:52.999+0000 7fa7532fc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:45:53.283 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:53 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:53.092+0000 7fa7532fc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:45:53.284 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:53 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:53.293+0000 7fa7532fc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:45:53.344 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T21:45:53.344 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:45:53.344 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "85326acc-dff0-11ee-95c8-87774f69a715", 2024-03-11T21:45:53.344 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:45:53.344 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:45:53.344 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:45:53.344 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:45:53.344 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:53.344 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:45:53.344 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:45:53.344 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:45:53.344 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:45:53.344 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:45:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:45:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:45:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-11T21:45:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:45:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:45:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:45:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:45:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:45:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:45:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:45:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:45:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:45:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:45:53.346 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:45:53.346 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:45:53.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:53.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:45:53.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:45:53.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:45:53.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:45:53.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:45:53.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:45:53.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:45:53.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:45:53.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:45:53.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:53.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:45:53.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:45:53.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:45:53.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:45:53.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:53.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:45:53.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:45:53.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:45:53.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:45:53.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:45:53.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:45:53.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:45:53.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:45:53.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:45:53.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:53.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:45:53.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:45:53.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:45:41.992595+0000", 2024-03-11T21:45:53.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:45:53.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:53.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:45:53.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:45:53.351 INFO:teuthology.orchestra.run.smithi002.stdout:mgr not available, waiting (2/15)... 2024-03-11T21:45:53.643 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:53 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:53.481+0000 7fa7532fc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:45:54.024 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:53 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:53.772+0000 7fa7532fc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:45:54.024 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:54 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:54.033+0000 7fa7532fc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:45:54.353 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:54 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:54.275+0000 7fa7532fc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:45:55.393 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:55.107+0000 7fa7532fc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:45:55.782 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:55.704+0000 7fa7532fc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:45:56.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:55 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/3568315503' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:45:56.144 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:55.808+0000 7fa7532fc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:45:56.144 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:45:56 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:45:56.036+0000 7fa7532fc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:45:56.577 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T21:45:56.577 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:45:56.577 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "85326acc-dff0-11ee-95c8-87774f69a715", 2024-03-11T21:45:56.577 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:45:56.577 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:45:56.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:45:56.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:45:56.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:56.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:45:56.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:45:56.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:45:56.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:45:56.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:45:56.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:45:56.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:45:56.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-11T21:45:56.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:45:56.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:45:56.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:45:56.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:45:56.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:56.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:45:56.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:45:56.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:45:56.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:45:56.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:45:56.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:45:56.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:45:56.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:45:56.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:56.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:45:56.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:45:56.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:45:56.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:45:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:45:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:45:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:45:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:45:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:45:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:45:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:45:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:45:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:45:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:45:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:45:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:45:56.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:45:56.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:45:56.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:45:56.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:45:56.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:45:56.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:45:56.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:56.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:45:56.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:45:56.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:45:41.992595+0000", 2024-03-11T21:45:56.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:45:56.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:45:56.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:45:56.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:45:56.581 INFO:teuthology.orchestra.run.smithi002.stdout:mgr not available, waiting (3/15)... 2024-03-11T21:45:57.393 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:56 smithi002 ceph-mon[25764]: Activating manager daemon y 2024-03-11T21:45:57.393 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:56 smithi002 ceph-mon[25764]: mgrmap e2: y(active, starting, since 0.00310654s) 2024-03-11T21:45:57.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:56 smithi002 ceph-mon[25764]: from='mgr.14100 172.21.15.2:0/2625766378' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:45:57.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:56 smithi002 ceph-mon[25764]: from='mgr.14100 172.21.15.2:0/2625766378' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:45:57.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:56 smithi002 ceph-mon[25764]: from='mgr.14100 172.21.15.2:0/2625766378' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:45:57.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:56 smithi002 ceph-mon[25764]: from='mgr.14100 172.21.15.2:0/2625766378' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:45:57.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:56 smithi002 ceph-mon[25764]: from='mgr.14100 172.21.15.2:0/2625766378' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:45:57.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:56 smithi002 ceph-mon[25764]: Manager daemon y is now available 2024-03-11T21:45:57.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:56 smithi002 ceph-mon[25764]: from='mgr.14100 172.21.15.2:0/2625766378' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:45:57.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:56 smithi002 ceph-mon[25764]: from='mgr.14100 172.21.15.2:0/2625766378' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:45:57.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:56 smithi002 ceph-mon[25764]: from='mgr.14100 172.21.15.2:0/2625766378' entity='mgr.y' 2024-03-11T21:45:57.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:56 smithi002 ceph-mon[25764]: from='mgr.14100 172.21.15.2:0/2625766378' entity='mgr.y' 2024-03-11T21:45:57.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:56 smithi002 ceph-mon[25764]: from='mgr.14100 172.21.15.2:0/2625766378' entity='mgr.y' 2024-03-11T21:45:58.393 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:45:58 smithi002 ceph-mon[25764]: mgrmap e3: y(active, since 1.00707s) 2024-03-11T21:46:00.393 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:00 smithi002 ceph-mon[25764]: mgrmap e4: y(active, since 3s) 2024-03-11T21:46:02.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:01 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1100118555' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:46:06.205 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T21:46:06.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:46:06.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "85326acc-dff0-11ee-95c8-87774f69a715", 2024-03-11T21:46:06.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:46:06.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:46:06.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:46:06.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:46:06.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:46:06.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:46:06.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:46:06.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:46:06.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:46:06.206 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:46:06.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:46:06.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:46:06.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-11T21:46:06.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:46:06.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:46:06.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:46:06.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:46:06.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:46:06.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:46:06.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:46:06.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:46:06.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:46:06.207 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:46:06.211 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:46:06.211 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:46:06.211 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:46:06.211 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:46:06.211 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:46:06.211 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:46:06.211 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:46:06.211 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:46:06.211 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:46:06.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:46:06.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:46:06.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:46:06.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:46:06.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:46:06.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:46:06.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:46:06.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:46:06.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:46:06.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:46:06.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:46:06.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:46:06.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:46:06.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:46:06.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:46:06.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:46:06.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:46:06.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:46:06.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:46:06.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:46:06.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:46:06.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:46:06.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:45:41.992595+0000", 2024-03-11T21:46:06.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:46:06.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:46:06.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:46:06.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:46:06.213 INFO:teuthology.orchestra.run.smithi002.stdout:mgr is available 2024-03-11T21:46:07.143 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:06 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/4227004725' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T21:46:07.143 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:06 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/4227004725' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T21:46:07.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T21:46:07.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T21:46:07.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout fsid = 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:46:07.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T21:46:07.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T21:46:07.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T21:46:07.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T21:46:07.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T21:46:07.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T21:46:07.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T21:46:07.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T21:46:07.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T21:46:07.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T21:46:07.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T21:46:07.437 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling cephadm module... 2024-03-11T21:46:08.529 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:08 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1500277054' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T21:46:08.529 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:08 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: ignoring --setuser ceph since I am not root 2024-03-11T21:46:08.529 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:08 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: ignoring --setgroup ceph since I am not root 2024-03-11T21:46:09.039 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:08 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:08.797+0000 7f9592de3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:46:09.039 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:08 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:08.893+0000 7f9592de3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:46:09.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:09 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1500277054' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T21:46:09.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:09 smithi002 ceph-mon[25764]: mgrmap e5: y(active, since 12s) 2024-03-11T21:46:09.893 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:09 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:09.576+0000 7f9592de3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:46:09.894 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:09 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:09.670+0000 7f9592de3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:46:09.894 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:09 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:09.817+0000 7f9592de3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:46:10.221 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:09 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:09.914+0000 7f9592de3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:46:10.221 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:10 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:09.999+0000 7f9592de3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:46:10.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:10 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/3638907802' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T21:46:10.472 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:46:10.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:10 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:10.269+0000 7f9592de3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:46:10.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:10 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:10.380+0000 7f9592de3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:46:10.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:10 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:10.468+0000 7f9592de3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:46:10.473 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T21:46:10.473 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:46:10.473 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T21:46:10.473 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T21:46:10.473 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:46:10.473 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for the mgr to restart... 2024-03-11T21:46:10.474 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr epoch 5... 2024-03-11T21:46:10.724 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:10 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:10.569+0000 7f9592de3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:46:11.143 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:10 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:10.755+0000 7f9592de3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:46:12.143 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:11 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:11.736+0000 7f9592de3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:46:12.143 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:11 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:11.869+0000 7f9592de3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:46:12.486 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:12 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:12.204+0000 7f9592de3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:46:12.486 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:12 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:12.296+0000 7f9592de3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:46:12.486 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:12 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:12.495+0000 7f9592de3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:46:12.893 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:12 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:12.682+0000 7f9592de3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:46:13.223 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:12 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:12.971+0000 7f9592de3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:46:13.223 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:13 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:13.232+0000 7f9592de3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:46:13.643 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:13 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:13.477+0000 7f9592de3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:46:14.643 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:14 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:14.301+0000 7f9592de3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:46:15.143 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:14 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:14.881+0000 7f9592de3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:46:15.143 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:14 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:14.981+0000 7f9592de3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:46:15.644 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:15.201+0000 7f9592de3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:46:15.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-mon[25764]: Active manager daemon y restarted 2024-03-11T21:46:15.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-mon[25764]: Activating manager daemon y 2024-03-11T21:46:15.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-mon[25764]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T21:46:15.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-mon[25764]: mgrmap e6: y(active, starting, since 0.00321083s) 2024-03-11T21:46:15.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:46:15.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:46:15.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:46:15.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:46:15.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:46:15.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-mon[25764]: Manager daemon y is now available 2024-03-11T21:46:15.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:15.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:15.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:15.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:15.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:15 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:16.500 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:16 smithi002 ceph-mon[25764]: Found migration_current of "None". Setting to last migration. 2024-03-11T21:46:16.500 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:16 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:16.500 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:16 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:46:16.500 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:16 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:46:16.500 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:16 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:16.500 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:16 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:16.500 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:16 smithi002 ceph-mon[25764]: mgrmap e7: y(active, since 1.00519s) 2024-03-11T21:46:16.770 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:46:16.770 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T21:46:16.770 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T21:46:16.770 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:46:16.770 INFO:teuthology.orchestra.run.smithi002.stdout:mgr epoch 5 is available 2024-03-11T21:46:16.770 INFO:teuthology.orchestra.run.smithi002.stdout:Setting orchestrator backend to cephadm... 2024-03-11T21:46:17.393 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:17 smithi002 ceph-mon[25764]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T21:46:17.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:17 smithi002 ceph-mon[25764]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T21:46:17.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:17 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:18.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:18 smithi002 ceph-mon[25764]: [11/Mar/2024:21:46:16] ENGINE Bus STARTING 2024-03-11T21:46:18.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:18 smithi002 ceph-mon[25764]: [11/Mar/2024:21:46:16] ENGINE Serving on http://172.21.15.2:8765 2024-03-11T21:46:18.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:18 smithi002 ceph-mon[25764]: [11/Mar/2024:21:46:16] ENGINE Serving on https://172.21.15.2:7150 2024-03-11T21:46:18.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:18 smithi002 ceph-mon[25764]: [11/Mar/2024:21:46:16] ENGINE Bus STARTED 2024-03-11T21:46:18.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:18 smithi002 ceph-mon[25764]: mgrmap e8: y(active, since 2s) 2024-03-11T21:46:18.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:18 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:18.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:18 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:19.264 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T21:46:19.264 INFO:teuthology.orchestra.run.smithi002.stdout:Generating ssh key... 2024-03-11T21:46:19.589 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:19 smithi002 ceph-mon[25764]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:46:20.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-mon[25764]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:46:20.644 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: Generating public/private rsa key pair. 2024-03-11T21:46:20.644 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: Your identification has been saved in /tmp/tmpeih__psw/key. 2024-03-11T21:46:20.644 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: Your public key has been saved in /tmp/tmpeih__psw/key.pub. 2024-03-11T21:46:20.644 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: The key fingerprint is: 2024-03-11T21:46:20.644 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: SHA256:JPRsjAt7DzqKaiyGIn+3dbG10/+g74hdNRsnnQb02sw ceph-85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:46:20.644 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: The key's randomart image is: 2024-03-11T21:46:20.644 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: +---[RSA 3072]----+ 2024-03-11T21:46:20.644 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: | . . | 2024-03-11T21:46:20.645 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: | . = . . | 2024-03-11T21:46:20.645 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: | . o * . . | 2024-03-11T21:46:20.645 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: | o = *..| 2024-03-11T21:46:20.645 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: | . + S . ...E+| 2024-03-11T21:46:20.645 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: | o o + o.o=| 2024-03-11T21:46:20.645 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: |o o o o o +. | 2024-03-11T21:46:20.645 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: |== ..... . o = o | 2024-03-11T21:46:20.645 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: |X.o. ... . +oo +| 2024-03-11T21:46:20.645 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: +----[SHA256]-----+ 2024-03-11T21:46:21.354 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:21 smithi002 ceph-mon[25764]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:46:21.354 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:21 smithi002 ceph-mon[25764]: Generating ssh key... 2024-03-11T21:46:21.354 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:21 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:21.354 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:21 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:22.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdBMyYbODIjLepB9noFgmk5sK3NBN57SJ1WoWZtD+lloX1PvQG+ktXt/L/snPTTY+z8j4HtiYi9cNGVFQM7Tyw6UhJ+JDWPv8pC2BDoILU2t8CY/FNyU2Lm3NEpPjrbDQgP+6+YKzk81O1t0HWA9TLd4Hip6Ehgxgas+CD+BGrLK5MLuDFgjBk3fCUdxBiOnl6OnjsjDJo23u4v4JnGwwTy6rb1x9VmZWGMpkhSM8VbgiZAHAqMBDyr9HCbEzI2Jg8V4G29dGtjl0z/Qckzy1D+rG3qkd8lXeW+1jp2sz6Xd7AItWbhvHDjD9qH+lOT3P1mqZgvgF7W09EDKBs8s/VXvoK+2yIAjWC41iExAYHbybrIpQ8qbFbRUy6INRX23r95Mu+FmZ9WC5juwhC4lWIWxfidexQwgL9IbebqOdWhNrDgXaXsdNwrisulsVsD8RPIOKvAa+6k//Tg6QfBdjMVzxuDEFF0h41dqIIeB/Kin13yw5N6FX93K+08aJBfOs= ceph-85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:46:22.102 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:46:22.103 INFO:teuthology.orchestra.run.smithi002.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T21:46:22.103 INFO:teuthology.orchestra.run.smithi002.stdout:Adding host smithi002... 2024-03-11T21:46:22.352 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:22 smithi002 ceph-mon[25764]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:46:23.393 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:23 smithi002 ceph-mon[25764]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi002", "addr": "172.21.15.2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:46:24.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:24 smithi002 ceph-mon[25764]: Deploying cephadm binary to smithi002 2024-03-11T21:46:27.650 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Added host 'smithi002' with addr '172.21.15.2' 2024-03-11T21:46:27.650 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying unmanaged mon service... 2024-03-11T21:46:28.328 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:28 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:28.329 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:28 smithi002 ceph-mon[25764]: Added host smithi002 2024-03-11T21:46:28.329 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:28 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:29.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T21:46:29.102 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying unmanaged mgr service... 2024-03-11T21:46:29.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:29 smithi002 ceph-mon[25764]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:46:29.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:29 smithi002 ceph-mon[25764]: Saving service mon spec with placement count:5 2024-03-11T21:46:29.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:29 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:31.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:31 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:31.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:31 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:31.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:31 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:31.111 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T21:46:32.143 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:32 smithi002 ceph-mon[25764]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:46:32.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:32 smithi002 ceph-mon[25764]: Saving service mgr spec with placement count:2 2024-03-11T21:46:32.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:32 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/3496495699' entity='client.admin' 2024-03-11T21:46:34.041 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:33 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:34.041 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:33 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:34.041 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:33 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:46:34.041 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:33 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' 2024-03-11T21:46:34.041 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:33 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi002", "caps": []}]: dispatch 2024-03-11T21:46:34.041 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:33 smithi002 ceph-mon[25764]: from='mgr.14120 172.21.15.2:0/567254072' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi002", "caps": []}]': finished 2024-03-11T21:46:34.041 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:33 smithi002 ceph-mon[25764]: Deploying daemon agent.smithi002 on smithi002 2024-03-11T21:46:34.041 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:33 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/2708762011' entity='client.admin' 2024-03-11T21:46:34.294 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling the dashboard module... 2024-03-11T21:46:35.329 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:35 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/2441126540' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T21:46:35.330 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:35 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: ignoring --setuser ceph since I am not root 2024-03-11T21:46:35.330 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:35 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: ignoring --setgroup ceph since I am not root 2024-03-11T21:46:35.857 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:35 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:35.606+0000 7fae2bb0e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:46:35.857 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:35 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:35.707+0000 7fae2bb0e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:46:36.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:36 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/2441126540' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T21:46:36.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:36 smithi002 ceph-mon[25764]: mgrmap e9: y(active, since 19s) 2024-03-11T21:46:36.611 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:36 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:36.378+0000 7fae2bb0e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:46:36.611 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:36 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:36.476+0000 7fae2bb0e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:46:36.611 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:36 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:36.620+0000 7fae2bb0e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:46:36.893 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:36 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:36.719+0000 7fae2bb0e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:46:36.894 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:36 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:36.805+0000 7fae2bb0e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:46:36.948 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:46:36.948 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T21:46:36.948 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:46:36.948 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T21:46:36.948 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T21:46:36.949 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:46:36.949 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for the mgr to restart... 2024-03-11T21:46:36.949 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr epoch 9... 2024-03-11T21:46:37.200 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:37 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/789607099' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T21:46:37.201 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:37 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:37.078+0000 7fae2bb0e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:46:37.201 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:37 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:37.194+0000 7fae2bb0e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:46:37.556 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:37 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:37.285+0000 7fae2bb0e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:46:37.556 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:37 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:37.384+0000 7fae2bb0e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:46:37.556 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:37 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:37.565+0000 7fae2bb0e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:46:38.894 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:38 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:38.552+0000 7fae2bb0e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:46:38.894 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:38 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:38.686+0000 7fae2bb0e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:46:39.307 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:39 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:39.023+0000 7fae2bb0e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:46:39.307 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:39 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:39.116+0000 7fae2bb0e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:46:39.308 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:39 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:39.316+0000 7fae2bb0e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:46:39.643 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:39 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:39.506+0000 7fae2bb0e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:46:40.048 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:39 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:39.796+0000 7fae2bb0e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:46:40.048 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:40 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:40.057+0000 7fae2bb0e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:46:40.394 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:40 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:40.299+0000 7fae2bb0e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:46:41.394 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:41 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:41.125+0000 7fae2bb0e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:46:42.018 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:41 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:41.704+0000 7fae2bb0e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:46:42.018 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:41 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:41.805+0000 7fae2bb0e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:46:42.018 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:46:42 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:46:42.027+0000 7fae2bb0e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:46:42.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:42 smithi002 ceph-mon[25764]: Active manager daemon y restarted 2024-03-11T21:46:42.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:42 smithi002 ceph-mon[25764]: Activating manager daemon y 2024-03-11T21:46:42.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:42 smithi002 ceph-mon[25764]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T21:46:42.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:42 smithi002 ceph-mon[25764]: mgrmap e10: y(active, starting, since 0.00296052s) 2024-03-11T21:46:42.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:46:42.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:46:42.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:46:42.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:46:42.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:46:42.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:42 smithi002 ceph-mon[25764]: Manager daemon y is now available 2024-03-11T21:46:42.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:42.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:42.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:43.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:43 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:46:43.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:43 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:46:43.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:43 smithi002 ceph-mon[25764]: [11/Mar/2024:21:46:42] ENGINE Bus STARTING 2024-03-11T21:46:43.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:43 smithi002 ceph-mon[25764]: [11/Mar/2024:21:46:42] ENGINE Serving on https://172.21.15.2:7150 2024-03-11T21:46:43.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:43 smithi002 ceph-mon[25764]: [11/Mar/2024:21:46:42] ENGINE Serving on http://172.21.15.2:8765 2024-03-11T21:46:43.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:43 smithi002 ceph-mon[25764]: [11/Mar/2024:21:46:42] ENGINE Bus STARTED 2024-03-11T21:46:43.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:43 smithi002 ceph-mon[25764]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T21:46:43.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:43 smithi002 ceph-mon[25764]: mgrmap e11: y(active, since 1.00563s) 2024-03-11T21:46:43.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:43 smithi002 ceph-mon[25764]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T21:46:43.647 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:46:43.647 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T21:46:43.647 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T21:46:43.647 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:46:43.647 INFO:teuthology.orchestra.run.smithi002.stdout:mgr epoch 9 is available 2024-03-11T21:46:43.647 INFO:teuthology.orchestra.run.smithi002.stdout:Generating a dashboard self-signed certificate... 2024-03-11T21:46:45.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:45 smithi002 ceph-mon[25764]: mgrmap e12: y(active, since 2s) 2024-03-11T21:46:45.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:45 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:45.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:45 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:46:45.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:45 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:45.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:45 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi002", "caps": []}]: dispatch 2024-03-11T21:46:45.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:45 smithi002 ceph-mon[25764]: Deploying daemon agent.smithi002 on smithi002 2024-03-11T21:46:45.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:45 smithi002 ceph-mon[25764]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:46:45.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:45 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:45.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:45 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:45.405 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T21:46:45.405 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial admin user... 2024-03-11T21:46:46.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:46 smithi002 ceph-mon[25764]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:46:46.893 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$0qlYHBnGCsAoBvx4czZPSuntTUj4TeQw.MRHv7igJ3z.QHxWWSZfO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710193606, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T21:46:46.894 INFO:teuthology.orchestra.run.smithi002.stdout:Fetching dashboard port number... 2024-03-11T21:46:47.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:47 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:48.191 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T21:46:48.200 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:46:48.200 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2024-03-11T21:46:48.207 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:46:48.208 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-03-11T21:46:48.208 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2024-03-11T21:46:48.208 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T21:46:48.210 INFO:teuthology.orchestra.run.smithi002.stdout:Ceph Dashboard is now available at: 2024-03-11T21:46:48.210 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:46:48.210 INFO:teuthology.orchestra.run.smithi002.stdout: URL: https://smithi002.front.sepia.ceph.com:8443/ 2024-03-11T21:46:48.210 INFO:teuthology.orchestra.run.smithi002.stdout: User: admin 2024-03-11T21:46:48.210 INFO:teuthology.orchestra.run.smithi002.stdout: Password: tojpfxqwir 2024-03-11T21:46:48.210 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:46:48.211 INFO:teuthology.orchestra.run.smithi002.stdout:Saving cluster configuration to /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config directory 2024-03-11T21:46:48.211 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling autotune for osd_memory_target 2024-03-11T21:46:48.446 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:48 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/819324704' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T21:46:49.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:49 smithi002 ceph-mon[25764]: mgrmap e13: y(active, since 6s) 2024-03-11T21:46:50.728 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T21:46:50.728 INFO:teuthology.orchestra.run.smithi002.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T21:46:50.728 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:46:50.729 INFO:teuthology.orchestra.run.smithi002.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:46:50.729 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:46:50.729 INFO:teuthology.orchestra.run.smithi002.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T21:46:50.729 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:46:50.729 INFO:teuthology.orchestra.run.smithi002.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T21:46:50.729 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:46:50.729 INFO:teuthology.orchestra.run.smithi002.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T21:46:50.729 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:46:50.729 INFO:teuthology.orchestra.run.smithi002.stdout: ceph telemetry on 2024-03-11T21:46:50.729 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:46:50.729 INFO:teuthology.orchestra.run.smithi002.stdout:For more information see: 2024-03-11T21:46:50.729 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:46:50.729 INFO:teuthology.orchestra.run.smithi002.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-11T21:46:50.729 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:46:50.730 INFO:teuthology.orchestra.run.smithi002.stdout:Bootstrap complete. 2024-03-11T21:46:50.775 INFO:tasks.cephadm:Fetching config... 2024-03-11T21:46:50.775 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T21:46:50.775 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T21:46:50.790 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T21:46:50.790 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T21:46:50.790 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T21:46:50.848 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T21:46:50.848 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T21:46:50.848 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T21:46:51.007 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T21:46:51.007 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T21:46:51.007 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T21:46:51.068 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T21:46:51.069 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdBMyYbODIjLepB9noFgmk5sK3NBN57SJ1WoWZtD+lloX1PvQG+ktXt/L/snPTTY+z8j4HtiYi9cNGVFQM7Tyw6UhJ+JDWPv8pC2BDoILU2t8CY/FNyU2Lm3NEpPjrbDQgP+6+YKzk81O1t0HWA9TLd4Hip6Ehgxgas+CD+BGrLK5MLuDFgjBk3fCUdxBiOnl6OnjsjDJo23u4v4JnGwwTy6rb1x9VmZWGMpkhSM8VbgiZAHAqMBDyr9HCbEzI2Jg8V4G29dGtjl0z/Qckzy1D+rG3qkd8lXeW+1jp2sz6Xd7AItWbhvHDjD9qH+lOT3P1mqZgvgF7W09EDKBs8s/VXvoK+2yIAjWC41iExAYHbybrIpQ8qbFbRUy6INRX23r95Mu+FmZ9WC5juwhC4lWIWxfidexQwgL9IbebqOdWhNrDgXaXsdNwrisulsVsD8RPIOKvAa+6k//Tg6QfBdjMVzxuDEFF0h41dqIIeB/Kin13yw5N6FX93K+08aJBfOs= ceph-85326acc-dff0-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T21:46:51.206 INFO:teuthology.orchestra.run.smithi002.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdBMyYbODIjLepB9noFgmk5sK3NBN57SJ1WoWZtD+lloX1PvQG+ktXt/L/snPTTY+z8j4HtiYi9cNGVFQM7Tyw6UhJ+JDWPv8pC2BDoILU2t8CY/FNyU2Lm3NEpPjrbDQgP+6+YKzk81O1t0HWA9TLd4Hip6Ehgxgas+CD+BGrLK5MLuDFgjBk3fCUdxBiOnl6OnjsjDJo23u4v4JnGwwTy6rb1x9VmZWGMpkhSM8VbgiZAHAqMBDyr9HCbEzI2Jg8V4G29dGtjl0z/Qckzy1D+rG3qkd8lXeW+1jp2sz6Xd7AItWbhvHDjD9qH+lOT3P1mqZgvgF7W09EDKBs8s/VXvoK+2yIAjWC41iExAYHbybrIpQ8qbFbRUy6INRX23r95Mu+FmZ9WC5juwhC4lWIWxfidexQwgL9IbebqOdWhNrDgXaXsdNwrisulsVsD8RPIOKvAa+6k//Tg6QfBdjMVzxuDEFF0h41dqIIeB/Kin13yw5N6FX93K+08aJBfOs= ceph-85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:46:51.220 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdBMyYbODIjLepB9noFgmk5sK3NBN57SJ1WoWZtD+lloX1PvQG+ktXt/L/snPTTY+z8j4HtiYi9cNGVFQM7Tyw6UhJ+JDWPv8pC2BDoILU2t8CY/FNyU2Lm3NEpPjrbDQgP+6+YKzk81O1t0HWA9TLd4Hip6Ehgxgas+CD+BGrLK5MLuDFgjBk3fCUdxBiOnl6OnjsjDJo23u4v4JnGwwTy6rb1x9VmZWGMpkhSM8VbgiZAHAqMBDyr9HCbEzI2Jg8V4G29dGtjl0z/Qckzy1D+rG3qkd8lXeW+1jp2sz6Xd7AItWbhvHDjD9qH+lOT3P1mqZgvgF7W09EDKBs8s/VXvoK+2yIAjWC41iExAYHbybrIpQ8qbFbRUy6INRX23r95Mu+FmZ9WC5juwhC4lWIWxfidexQwgL9IbebqOdWhNrDgXaXsdNwrisulsVsD8RPIOKvAa+6k//Tg6QfBdjMVzxuDEFF0h41dqIIeB/Kin13yw5N6FX93K+08aJBfOs= ceph-85326acc-dff0-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T21:46:51.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:51 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/4173602770' entity='client.admin' 2024-03-11T21:46:51.264 INFO:teuthology.orchestra.run.smithi122.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdBMyYbODIjLepB9noFgmk5sK3NBN57SJ1WoWZtD+lloX1PvQG+ktXt/L/snPTTY+z8j4HtiYi9cNGVFQM7Tyw6UhJ+JDWPv8pC2BDoILU2t8CY/FNyU2Lm3NEpPjrbDQgP+6+YKzk81O1t0HWA9TLd4Hip6Ehgxgas+CD+BGrLK5MLuDFgjBk3fCUdxBiOnl6OnjsjDJo23u4v4JnGwwTy6rb1x9VmZWGMpkhSM8VbgiZAHAqMBDyr9HCbEzI2Jg8V4G29dGtjl0z/Qckzy1D+rG3qkd8lXeW+1jp2sz6Xd7AItWbhvHDjD9qH+lOT3P1mqZgvgF7W09EDKBs8s/VXvoK+2yIAjWC41iExAYHbybrIpQ8qbFbRUy6INRX23r95Mu+FmZ9WC5juwhC4lWIWxfidexQwgL9IbebqOdWhNrDgXaXsdNwrisulsVsD8RPIOKvAa+6k//Tg6QfBdjMVzxuDEFF0h41dqIIeB/Kin13yw5N6FX93K+08aJBfOs= ceph-85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:46:51.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdBMyYbODIjLepB9noFgmk5sK3NBN57SJ1WoWZtD+lloX1PvQG+ktXt/L/snPTTY+z8j4HtiYi9cNGVFQM7Tyw6UhJ+JDWPv8pC2BDoILU2t8CY/FNyU2Lm3NEpPjrbDQgP+6+YKzk81O1t0HWA9TLd4Hip6Ehgxgas+CD+BGrLK5MLuDFgjBk3fCUdxBiOnl6OnjsjDJo23u4v4JnGwwTy6rb1x9VmZWGMpkhSM8VbgiZAHAqMBDyr9HCbEzI2Jg8V4G29dGtjl0z/Qckzy1D+rG3qkd8lXeW+1jp2sz6Xd7AItWbhvHDjD9qH+lOT3P1mqZgvgF7W09EDKBs8s/VXvoK+2yIAjWC41iExAYHbybrIpQ8qbFbRUy6INRX23r95Mu+FmZ9WC5juwhC4lWIWxfidexQwgL9IbebqOdWhNrDgXaXsdNwrisulsVsD8RPIOKvAa+6k//Tg6QfBdjMVzxuDEFF0h41dqIIeB/Kin13yw5N6FX93K+08aJBfOs= ceph-85326acc-dff0-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T21:46:51.325 INFO:teuthology.orchestra.run.smithi165.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdBMyYbODIjLepB9noFgmk5sK3NBN57SJ1WoWZtD+lloX1PvQG+ktXt/L/snPTTY+z8j4HtiYi9cNGVFQM7Tyw6UhJ+JDWPv8pC2BDoILU2t8CY/FNyU2Lm3NEpPjrbDQgP+6+YKzk81O1t0HWA9TLd4Hip6Ehgxgas+CD+BGrLK5MLuDFgjBk3fCUdxBiOnl6OnjsjDJo23u4v4JnGwwTy6rb1x9VmZWGMpkhSM8VbgiZAHAqMBDyr9HCbEzI2Jg8V4G29dGtjl0z/Qckzy1D+rG3qkd8lXeW+1jp2sz6Xd7AItWbhvHDjD9qH+lOT3P1mqZgvgF7W09EDKBs8s/VXvoK+2yIAjWC41iExAYHbybrIpQ8qbFbRUy6INRX23r95Mu+FmZ9WC5juwhC4lWIWxfidexQwgL9IbebqOdWhNrDgXaXsdNwrisulsVsD8RPIOKvAa+6k//Tg6QfBdjMVzxuDEFF0h41dqIIeB/Kin13yw5N6FX93K+08aJBfOs= ceph-85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:46:51.340 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T21:46:51.733 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:52.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:52.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:52.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:52.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:52.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:52.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:52.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:52.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:52.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:52.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:52.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:52.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:53.334 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:53 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:54.212 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T21:46:54.212 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T21:46:54.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:54 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/2053957880' entity='client.admin' 2024-03-11T21:46:54.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:54 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:54.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:54 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:56.118 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:57.539 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:57 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:57.539 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:57 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:58.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:58 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:58.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:58 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:58.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:58 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:59.247 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi122 2024-03-11T21:46:59.248 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-11T21:46:59.248 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.conf 2024-03-11T21:46:59.265 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-11T21:46:59.265 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:46:59.322 INFO:tasks.cephadm:Adding host smithi122 to orchestrator... 2024-03-11T21:46:59.323 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph orch host add smithi122 2024-03-11T21:46:59.567 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:59.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:59 smithi002 ceph-mon[25764]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:46:59.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:59 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:59.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:59 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:46:59.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:59 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:46:59.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:59 smithi002 ceph-mon[25764]: Updating smithi002:/etc/ceph/ceph.conf 2024-03-11T21:46:59.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:59 smithi002 ceph-mon[25764]: Updating smithi002:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:46:59.794 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:59 smithi002 ceph-mon[25764]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:46:59.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:59 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:59.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:59 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:46:59.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:46:59 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:00.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:00 smithi002 ceph-mon[25764]: Updating smithi002:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:47:03.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:03 smithi002 ceph-mon[25764]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:47:03.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:03 smithi002 ceph-mon[25764]: Deploying cephadm binary to smithi122 2024-03-11T21:47:03.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:03 smithi002 ceph-mon[25764]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:05.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:05 smithi002 ceph-mon[25764]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:05.673 INFO:teuthology.orchestra.run.smithi002.stdout:Added host 'smithi122' with addr '172.21.15.122' 2024-03-11T21:47:06.307 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T21:47:06.551 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:06.918 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:06 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:06.918 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:06 smithi002 ceph-mon[25764]: Added host smithi122 2024-03-11T21:47:06.918 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:06 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:06.918 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:06 smithi002 ceph-mon[25764]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:06.918 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:06 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:07.948 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:07 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:07.948 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:07 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:07.948 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:07 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:47:07.948 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:07 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:07.948 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:07 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:07.948 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:07 smithi002 ceph-mon[25764]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-11T21:47:07.948 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:07 smithi002 ceph-mon[25764]: Updating smithi122:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:47:07.949 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:07 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:07.949 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:07 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:07.949 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:07 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi122", "caps": []}]: dispatch 2024-03-11T21:47:07.949 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:07 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi122", "caps": []}]': finished 2024-03-11T21:47:08.419 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:47:08.419 INFO:teuthology.orchestra.run.smithi002.stdout:[{"addr": "172.21.15.2", "hostname": "smithi002", "labels": [], "status": ""}, {"addr": "172.21.15.122", "hostname": "smithi122", "labels": [], "status": ""}] 2024-03-11T21:47:08.977 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:08 smithi002 ceph-mon[25764]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:47:08.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:08 smithi002 ceph-mon[25764]: Updating smithi122:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:47:08.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:08 smithi002 ceph-mon[25764]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:09.003 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi165 2024-03-11T21:47:09.003 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T21:47:09.003 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.conf 2024-03-11T21:47:09.023 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T21:47:09.023 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:47:09.081 INFO:tasks.cephadm:Adding host smithi165 to orchestrator... 2024-03-11T21:47:09.081 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph orch host add smithi165 2024-03-11T21:47:09.315 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:10.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:09 smithi002 ceph-mon[25764]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:47:10.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:09 smithi002 ceph-mon[25764]: Deploying daemon agent.smithi122 on smithi122 2024-03-11T21:47:11.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:11.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:11.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:11.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:11.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:11.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:11.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:11.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:11.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:11.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:11.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:11.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:10 smithi002 ceph-mon[25764]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:11.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:11.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:11.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:12.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:12 smithi002 ceph-mon[25764]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:47:13.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:13 smithi002 ceph-mon[25764]: Deploying cephadm binary to smithi165 2024-03-11T21:47:13.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:13 smithi002 ceph-mon[25764]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:13.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:13 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:14.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:14 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:14.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:14 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:14.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:14 smithi002 ceph-mon[25764]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:14.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:14 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:14.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:14 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:14.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:14 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:15.647 INFO:teuthology.orchestra.run.smithi002.stdout:Added host 'smithi165' with addr '172.21.15.165' 2024-03-11T21:47:16.209 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T21:47:16.454 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:16.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:16.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:16 smithi002 ceph-mon[25764]: Added host smithi165 2024-03-11T21:47:16.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:16.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:16 smithi002 ceph-mon[25764]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:16.856 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:17.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:17 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:17.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:17 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:17.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:17 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:47:17.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:17 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:17.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:17 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:17.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:17 smithi002 ceph-mon[25764]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-11T21:47:17.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:17 smithi002 ceph-mon[25764]: Updating smithi165:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:47:17.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:17 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:17.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:17 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:17.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:17 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]: dispatch 2024-03-11T21:47:17.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:17 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]': finished 2024-03-11T21:47:18.315 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:47:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:[{"addr": "172.21.15.2", "hostname": "smithi002", "labels": [], "status": ""}, {"addr": "172.21.15.122", "hostname": "smithi122", "labels": [], "status": ""}, {"addr": "172.21.15.165", "hostname": "smithi165", "labels": [], "status": ""}] 2024-03-11T21:47:18.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:18 smithi002 ceph-mon[25764]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:47:18.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:18 smithi002 ceph-mon[25764]: Updating smithi165:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:47:18.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:18 smithi002 ceph-mon[25764]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:18.928 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T21:47:18.928 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T21:47:19.169 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:19.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:19 smithi002 ceph-mon[25764]: Deploying daemon agent.smithi165 on smithi165 2024-03-11T21:47:19.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:19 smithi002 ceph-mon[25764]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:47:19.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:19.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:20.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:20 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:20.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:20 smithi002 ceph-mon[25764]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:20.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:20 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:20.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:20 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:21.804 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2024-03-11T21:47:22.137 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:21 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/2091315693' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T21:47:22.447 INFO:tasks.cephadm:Adding mon.a on smithi002 2024-03-11T21:47:22.447 INFO:tasks.cephadm:Adding mon.b on smithi122 2024-03-11T21:47:22.447 INFO:tasks.cephadm:Adding mon.c on smithi165 2024-03-11T21:47:22.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi002:172.21.15.2=a;smithi122:172.21.15.122=b;smithi165:172.21.15.165=c' 2024-03-11T21:47:22.693 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:47:23.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:22 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/2091315693' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T21:47:23.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:22 smithi002 ceph-mon[25764]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:47:23.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:22 smithi002 ceph-mon[25764]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:23.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:22 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:23.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:22 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:23.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:22 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:24.380 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled mon update... 2024-03-11T21:47:24.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:24 smithi002 ceph-mon[25764]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:24.967 DEBUG:teuthology.orchestra.run.smithi122:mon.b> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T21:47:24.970 DEBUG:teuthology.orchestra.run.smithi165:mon.c> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T21:47:24.973 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:47:24.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:47:25.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:25 smithi002 ceph-mon[25764]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi122:172.21.15.122=b;smithi165:172.21.15.165=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:47:25.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:25 smithi002 ceph-mon[25764]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi122:172.21.15.122=b;smithi165:172.21.15.165=c;count:3 2024-03-11T21:47:25.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:25.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:25.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:25.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:25.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:25.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:47:25.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:25.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:25 smithi002 ceph-mon[25764]: Deploying daemon mon.c on smithi165 2024-03-11T21:47:25.756 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:47:26.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:26 smithi002 ceph-mon[25764]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:28.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:28 smithi002 ceph-mon[25764]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:28.979 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T21:47:28.979 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":1,"fsid":"85326acc-dff0-11ee-95c8-87774f69a715","modified":"2024-03-11T21:45:38.990712Z","created":"2024-03-11T21:45:38.990712Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T21:47:28.981 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 1 2024-03-11T21:47:29.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:29 smithi002 ceph-mon[25764]: from='client.? 172.21.15.165:0/2415926802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:47:30.789 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:47:30.789 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:47:31.034 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:47:34.534 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:47:34.534 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: mon.a calling monitor election 2024-03-11T21:47:34.534 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:34.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:34.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:34.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:34.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: mon.c calling monitor election 2024-03-11T21:47:34.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:34.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:34.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:34.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:34.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:34.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:47:34.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:47:34.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: fsmap 2024-03-11T21:47:34.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:47:34.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: mgrmap e13: y(active, since 52s) 2024-03-11T21:47:34.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: overall HEALTH_OK 2024-03-11T21:47:34.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:34.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:34.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:34.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:34.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:34.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:34.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:34.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:34.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:34.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:47:34.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:34 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:34.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:34 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:35.662 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T21:47:35.662 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":2,"fsid":"85326acc-dff0-11ee-95c8-87774f69a715","modified":"2024-03-11T21:47:29.387490Z","created":"2024-03-11T21:45:38.990712Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T21:47:35.664 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 2 2024-03-11T21:47:35.832 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:35 smithi165 ceph-mon[27942]: Deploying daemon mon.b on smithi122 2024-03-11T21:47:35.832 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:35 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:35.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:35 smithi002 ceph-mon[25764]: Deploying daemon mon.b on smithi122 2024-03-11T21:47:35.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:35 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:36.832 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:36 smithi165 ceph-mon[27942]: from='client.? 172.21.15.165:0/1515218049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:47:36.832 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:36 smithi165 ceph-mon[27942]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:36 smithi002 ceph-mon[25764]: from='client.? 172.21.15.165:0/1515218049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:47:36.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:36 smithi002 ceph-mon[25764]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:37.426 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:47:37.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:47:37.670 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:47:38.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:38 smithi165 ceph-mon[27942]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.367 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 podman[27547]: 2024-03-11 21:47:38.124540441 +0000 UTC m=+0.114994770 container create 40175a808c6d8033f168b70b85cec240b8f7c653ca95718a2db5774e3d620af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-11T21:47:38.368 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 podman[27547]: 2024-03-11 21:47:38.377090367 +0000 UTC m=+0.367544710 container init 40175a808c6d8033f168b70b85cec240b8f7c653ca95718a2db5774e3d620af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:47:38.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:38 smithi002 ceph-mon[25764]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 podman[27547]: 2024-03-11 21:47:38.382438035 +0000 UTC m=+0.372892364 container start 40175a808c6d8033f168b70b85cec240b8f7c653ca95718a2db5774e3d620af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:47:38.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T21:47:38.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 7 2024-03-11T21:47:38.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pidfile_write: ignore empty --pid-file 2024-03-11T21:47:38.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: load: jerasure load: lrc 2024-03-11T21:47:38.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: RocksDB version: 7.9.2 2024-03-11T21:47:38.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Git sha 0 2024-03-11T21:47:38.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T21:47:38.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: DB SUMMARY 2024-03-11T21:47:38.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: DB Session ID: 0L7LN432DEWQRLIFFUUD 2024-03-11T21:47:38.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: CURRENT file: CURRENT 2024-03-11T21:47:38.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T21:47:38.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T21:47:38.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-11T21:47:38.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-11T21:47:38.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.error_if_exists: 0 2024-03-11T21:47:38.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.create_if_missing: 0 2024-03-11T21:47:38.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.paranoid_checks: 1 2024-03-11T21:47:38.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T21:47:38.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T21:47:38.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T21:47:38.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.env: 0x5614e907cae0 2024-03-11T21:47:38.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T21:47:38.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.info_log: 0x5614ea2af8c0 2024-03-11T21:47:38.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T21:47:38.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.statistics: (nil) 2024-03-11T21:47:38.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.use_fsync: 0 2024-03-11T21:47:38.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_log_file_size: 0 2024-03-11T21:47:38.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T21:47:38.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T21:47:38.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T21:47:38.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T21:47:38.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.allow_fallocate: 1 2024-03-11T21:47:38.678 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T21:47:38.678 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T21:47:38.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.use_direct_reads: 0 2024-03-11T21:47:38.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T21:47:38.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T21:47:38.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.db_log_dir: 2024-03-11T21:47:38.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.wal_dir: 2024-03-11T21:47:38.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T21:47:38.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T21:47:38.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T21:47:38.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T21:47:38.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T21:47:38.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T21:47:38.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T21:47:38.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T21:47:38.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.write_buffer_manager: 0x5614ea283400 2024-03-11T21:47:38.679 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T21:47:38.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T21:47:38.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T21:47:38.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T21:47:38.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T21:47:38.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T21:47:38.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T21:47:38.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T21:47:38.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.unordered_write: 0 2024-03-11T21:47:38.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T21:47:38.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T21:47:38.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T21:47:38.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T21:47:38.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.row_cache: None 2024-03-11T21:47:38.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.wal_filter: None 2024-03-11T21:47:38.680 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T21:47:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T21:47:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.two_write_queues: 0 2024-03-11T21:47:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T21:47:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.wal_compression: 0 2024-03-11T21:47:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.atomic_flush: 0 2024-03-11T21:47:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T21:47:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T21:47:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T21:47:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.log_readahead_size: 0 2024-03-11T21:47:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T21:47:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T21:47:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T21:47:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T21:47:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T21:47:38.681 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T21:47:38.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T21:47:38.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_background_jobs: 2 2024-03-11T21:47:38.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_background_compactions: -1 2024-03-11T21:47:38.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_subcompactions: 1 2024-03-11T21:47:38.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T21:47:38.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T21:47:38.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T21:47:38.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T21:47:38.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T21:47:38.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T21:47:38.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T21:47:38.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T21:47:38.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_open_files: -1 2024-03-11T21:47:38.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T21:47:38.683 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T21:47:38.683 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T21:47:38.683 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T21:47:38.683 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_background_flushes: -1 2024-03-11T21:47:38.683 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Compression algorithms supported: 2024-03-11T21:47:38.683 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T21:47:38.683 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: kZSTD supported: 0 2024-03-11T21:47:38.683 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: kXpressCompression supported: 0 2024-03-11T21:47:38.683 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T21:47:38.683 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: kZlibCompression supported: 1 2024-03-11T21:47:38.683 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: kSnappyCompression supported: 1 2024-03-11T21:47:38.683 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: kLZ4Compression supported: 1 2024-03-11T21:47:38.683 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: kBZip2Compression supported: 0 2024-03-11T21:47:38.683 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T21:47:38.683 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T21:47:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-11T21:47:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T21:47:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T21:47:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.merge_operator: 2024-03-11T21:47:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compaction_filter: None 2024-03-11T21:47:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T21:47:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T21:47:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T21:47:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T21:47:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5614ea2af9c0) 2024-03-11T21:47:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout: cache_index_and_filter_blocks: 1 2024-03-11T21:47:38.684 INFO:journalctl@ceph.mon.b.smithi122.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T21:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T21:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout: pin_top_level_index_and_filter: 1 2024-03-11T21:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout: index_type: 0 2024-03-11T21:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout: data_block_index_type: 0 2024-03-11T21:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout: index_shortening: 1 2024-03-11T21:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T21:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout: checksum: 4 2024-03-11T21:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout: no_block_cache: 0 2024-03-11T21:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache: 0x5614ea2b3090 2024-03-11T21:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache_name: BinnedLRUCache 2024-03-11T21:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache_options: 2024-03-11T21:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout: capacity : 536870912 2024-03-11T21:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout: num_shard_bits : 4 2024-03-11T21:47:38.685 INFO:journalctl@ceph.mon.b.smithi122.stdout: strict_capacity_limit : 0 2024-03-11T21:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: high_pri_pool_ratio: 0.000 2024-03-11T21:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache_compressed: (nil) 2024-03-11T21:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: persistent_cache: (nil) 2024-03-11T21:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_size: 4096 2024-03-11T21:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_size_deviation: 10 2024-03-11T21:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_restart_interval: 16 2024-03-11T21:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: index_block_restart_interval: 1 2024-03-11T21:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: metadata_block_size: 4096 2024-03-11T21:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: partition_filters: 0 2024-03-11T21:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: use_delta_encoding: 1 2024-03-11T21:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: filter_policy: bloomfilter 2024-03-11T21:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: whole_key_filtering: 1 2024-03-11T21:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: verify_compression: 0 2024-03-11T21:47:38.686 INFO:journalctl@ceph.mon.b.smithi122.stdout: read_amp_bytes_per_bit: 0 2024-03-11T21:47:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout: format_version: 5 2024-03-11T21:47:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout: enable_index_compression: 1 2024-03-11T21:47:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_align: 0 2024-03-11T21:47:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout: max_auto_readahead_size: 262144 2024-03-11T21:47:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout: prepopulate_block_cache: 0 2024-03-11T21:47:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout: initial_auto_readahead_size: 8192 2024-03-11T21:47:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T21:47:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T21:47:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T21:47:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compression: NoCompression 2024-03-11T21:47:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T21:47:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T21:47:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T21:47:38.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.num_levels: 7 2024-03-11T21:47:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T21:47:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T21:47:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T21:47:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T21:47:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T21:47:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T21:47:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T21:47:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:47:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T21:47:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T21:47:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:47:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:47:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T21:47:38.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T21:47:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T21:47:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T21:47:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:47:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:47:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T21:47:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T21:47:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:47:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T21:47:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T21:47:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T21:47:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T21:47:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T21:47:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T21:47:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T21:47:38.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T21:47:38.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T21:47:38.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T21:47:38.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T21:47:38.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T21:47:38.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T21:47:38.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T21:47:38.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T21:47:38.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T21:47:38.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T21:47:38.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T21:47:38.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T21:47:38.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T21:47:38.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T21:47:38.690 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T21:47:38.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T21:47:38.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T21:47:38.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T21:47:38.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T21:47:38.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T21:47:38.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T21:47:38.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T21:47:38.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T21:47:38.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T21:47:38.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T21:47:38.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.table_properties_collectors: 2024-03-11T21:47:38.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.inplace_update_support: 0 2024-03-11T21:47:38.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T21:47:38.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T21:47:38.691 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T21:47:38.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T21:47:38.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.bloom_locality: 0 2024-03-11T21:47:38.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.max_successive_merges: 0 2024-03-11T21:47:38.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T21:47:38.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T21:47:38.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T21:47:38.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T21:47:38.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.ttl: 2592000 2024-03-11T21:47:38.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T21:47:38.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T21:47:38.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T21:47:38.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.enable_blob_files: false 2024-03-11T21:47:38.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.min_blob_size: 0 2024-03-11T21:47:38.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T21:47:38.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T21:47:38.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T21:47:38.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T21:47:38.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T21:47:38.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T21:47:38.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T21:47:38.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T21:47:38.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T21:47:38.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T21:47:38.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a10bccb9-6f25-49c2-8c07-9201b1fe1e3d 2024-03-11T21:47:38.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710193658413493, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T21:47:38.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T21:47:38.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710193658413999, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710193658, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a10bccb9-6f25-49c2-8c07-9201b1fe1e3d", "db_session_id": "0L7LN432DEWQRLIFFUUD", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T21:47:38.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710193658414097, "job": 1, "event": "recovery_finished"} 2024-03-11T21:47:38.693 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T21:47:38.694 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T21:47:38.694 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5614ea3a4000 2024-03-11T21:47:38.694 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: DB pointer 0x5614ea38e000 2024-03-11T21:47:38.694 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T21:47:38.694 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T21:47:38.694 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** DB Stats ** 2024-03-11T21:47:38.694 INFO:journalctl@ceph.mon.b.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:47:38.694 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T21:47:38.694 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:47:38.694 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:47:38.694 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T21:47:38.694 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:47:38.694 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:47:38.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-03-11T21:47:38.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** Compaction Stats [default] ** 2024-03-11T21:47:38.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:47:38.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T21:47:38.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:47:38.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:47:38.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:47:38.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-03-11T21:47:38.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** Compaction Stats [default] ** 2024-03-11T21:47:38.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:47:38.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T21:47:38.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:47:38.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-03-11T21:47:38.695 INFO:journalctl@ceph.mon.b.smithi122.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T21:47:38.696 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-03-11T21:47:38.696 INFO:journalctl@ceph.mon.b.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:47:38.696 INFO:journalctl@ceph.mon.b.smithi122.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T21:47:38.696 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T21:47:38.696 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T21:47:38.696 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T21:47:38.696 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T21:47:38.696 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:47:38.696 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:47:38.696 INFO:journalctl@ceph.mon.b.smithi122.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T21:47:38.696 INFO:journalctl@ceph.mon.b.smithi122.stdout: Block cache BinnedLRUCache@0x5614ea2b3090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-11T21:47:38.696 INFO:journalctl@ceph.mon.b.smithi122.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T21:47:38.696 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-03-11T21:47:38.696 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T21:47:38.697 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-11T21:47:38.697 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: using public_addr v2:172.21.15.122:0/0 -> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-03-11T21:47:38.697 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: starting mon.b rank -1 at public addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] at bind addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:47:38.697 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.b@-1(???) e0 preinit fsid 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:47:38.697 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.b@-1(synchronizing).mds e1 new map 2024-03-11T21:47:38.697 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-11T21:47:38.697 INFO:journalctl@ceph.mon.b.smithi122.stdout: e1 2024-03-11T21:47:38.697 INFO:journalctl@ceph.mon.b.smithi122.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T21:47:38.697 INFO:journalctl@ceph.mon.b.smithi122.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T21:47:38.697 INFO:journalctl@ceph.mon.b.smithi122.stdout: legacy client fscid: -1 2024-03-11T21:47:38.697 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-03-11T21:47:38.697 INFO:journalctl@ceph.mon.b.smithi122.stdout: No filesystems configured 2024-03-11T21:47:38.697 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T21:47:38.697 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T21:47:38.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T21:47:38.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T21:47:38.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T21:47:38.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T21:47:38.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T21:47:38.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:47:38.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:47:38.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:47:38.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Added host smithi122 2024-03-11T21:47:38.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:38.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:47:38.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:38.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:38.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-11T21:47:38.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Updating smithi122:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:47:38.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi122", "caps": []}]: dispatch 2024-03-11T21:47:38.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi122", "caps": []}]': finished 2024-03-11T21:47:38.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:47:38.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Updating smithi122:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:47:38.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:47:38.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Deploying daemon agent.smithi122 on smithi122 2024-03-11T21:47:38.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:38.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:38.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:38.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:38.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:38.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:38.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:47:38.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Deploying cephadm binary to smithi165 2024-03-11T21:47:38.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.701 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Added host smithi165 2024-03-11T21:47:38.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:38.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:47:38.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:38.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:38.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-11T21:47:38.702 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Updating smithi165:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:47:38.704 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.704 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.704 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]: dispatch 2024-03-11T21:47:38.704 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]': finished 2024-03-11T21:47:38.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:47:38.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Updating smithi165:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:47:38.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Deploying daemon agent.smithi165 on smithi165 2024-03-11T21:47:38.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:47:38.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:38.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:38.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:38.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:38.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:38.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:38.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/2091315693' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T21:47:38.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/2091315693' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T21:47:38.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:47:38.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi122:172.21.15.122=b;smithi165:172.21.15.165=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:47:38.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi122:172.21.15.122=b;smithi165:172.21.15.165=c;count:3 2024-03-11T21:47:38.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:38.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:38.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:38.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:47:38.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:38.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Deploying daemon mon.c on smithi165 2024-03-11T21:47:38.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='client.? 172.21.15.165:0/2415926802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:47:38.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:47:38.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.a calling monitor election 2024-03-11T21:47:38.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:38.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:38.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:38.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.c calling monitor election 2024-03-11T21:47:38.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:38.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:38.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:38.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:47:38.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:47:38.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: fsmap 2024-03-11T21:47:38.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:47:38.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mgrmap e13: y(active, since 52s) 2024-03-11T21:47:38.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: overall HEALTH_OK 2024-03-11T21:47:38.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:38.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:47:38.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:38.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: Deploying daemon mon.b on smithi122 2024-03-11T21:47:38.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:38.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: from='client.? 172.21.15.165:0/1515218049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:47:38.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:38.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 ceph-mon[27585]: mon.b@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-11T21:47:38.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 bash[27547]: 40175a808c6d8033f168b70b85cec240b8f7c653ca95718a2db5774e3d620af5 2024-03-11T21:47:38.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:38 smithi122 systemd[1]: Started Ceph mon.b for 85326acc-dff0-11ee-95c8-87774f69a715. 2024-03-11T21:47:43.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:47:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: mon.c calling monitor election 2024-03-11T21:47:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: mon.a calling monitor election 2024-03-11T21:47:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:47:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:47:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: fsmap 2024-03-11T21:47:43.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:47:43.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: mgrmap e13: y(active, since 61s) 2024-03-11T21:47:43.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T21:47:43.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T21:47:43.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T21:47:43.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: mon.b (rank 2) addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] is down (out of quorum) 2024-03-11T21:47:43.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:43.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:43.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:43.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:43.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:43 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:43.832 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:47:43.832 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: mon.c calling monitor election 2024-03-11T21:47:43.832 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: mon.a calling monitor election 2024-03-11T21:47:43.832 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:43.832 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:43.832 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:43.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:43.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:43.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:43.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:43.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:43.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:43.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:47:43.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:47:43.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: fsmap 2024-03-11T21:47:43.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:47:43.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: mgrmap e13: y(active, since 61s) 2024-03-11T21:47:43.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T21:47:43.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T21:47:43.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T21:47:43.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: mon.b (rank 2) addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] is down (out of quorum) 2024-03-11T21:47:43.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:43.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:43.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:43.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:43.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:43 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:43.909 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T21:47:43.909 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":3,"fsid":"85326acc-dff0-11ee-95c8-87774f69a715","modified":"2024-03-11T21:47:38.438289Z","created":"2024-03-11T21:45:38.990712Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T21:47:43.910 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 3 2024-03-11T21:47:44.531 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: Updating smithi002:/etc/ceph/ceph.conf 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='client.? 172.21.15.165:0/2748596322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: Updating smithi122:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: Updating smithi002:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: Updating smithi165:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.532 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.533 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.533 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.533 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.533 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.533 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.533 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:47:44.533 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:47:44.533 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:44.533 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:44 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:44.557 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T21:47:44.557 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T21:47:44.803 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:44.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:44.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:44.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:44.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: Updating smithi002:/etc/ceph/ceph.conf 2024-03-11T21:47:44.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-11T21:47:44.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-11T21:47:44.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='client.? 172.21.15.165:0/2748596322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:47:44.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: Updating smithi122:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:47:44.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: Updating smithi002:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:47:44.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: Updating smithi165:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:47:44.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:44.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:44.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:47:44.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:47:44.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:44.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:44 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:45.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:47:45.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: mon.c calling monitor election 2024-03-11T21:47:45.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: mon.a calling monitor election 2024-03-11T21:47:45.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:45.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:47:45.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:45.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:45.708 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:45.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:45.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:45.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:45.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:45.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T21:47:45.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:47:45.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: fsmap 2024-03-11T21:47:45.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:47:45.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: mgrmap e13: y(active, since 61s) 2024-03-11T21:47:45.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T21:47:45.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T21:47:45.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T21:47:45.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: mon.b (rank 2) addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] is down (out of quorum) 2024-03-11T21:47:45.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:45.709 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:45.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:45.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:45.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:45.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:45.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:45.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:45.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:45.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:45.710 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: Updating smithi002:/etc/ceph/ceph.conf 2024-03-11T21:47:45.713 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-11T21:47:45.713 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-11T21:47:45.713 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='client.? 172.21.15.165:0/2748596322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:47:45.713 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: Updating smithi122:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:47:45.713 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: Updating smithi002:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:47:45.713 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: Updating smithi165:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:47:45.713 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:45.714 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:45.714 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:45.714 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:45.714 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:45.714 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:45.714 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:45.714 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:45.714 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:47:45.714 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:47:45.714 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:45.714 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:45 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:46.576 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:46 smithi122 ceph-mon[27585]: mon.b calling monitor election 2024-03-11T21:47:46.576 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:46 smithi122 ceph-mon[27585]: mon.b calling monitor election 2024-03-11T21:47:46.577 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:46 smithi122 ceph-mon[27585]: mon.c calling monitor election 2024-03-11T21:47:46.577 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:46 smithi122 ceph-mon[27585]: mon.a calling monitor election 2024-03-11T21:47:46.577 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:46 smithi122 ceph-mon[27585]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T21:47:46.577 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:46 smithi122 ceph-mon[27585]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:47:46.577 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:46 smithi122 ceph-mon[27585]: fsmap 2024-03-11T21:47:46.577 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:46 smithi122 ceph-mon[27585]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:47:46.577 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:46 smithi122 ceph-mon[27585]: mgrmap e13: y(active, since 63s) 2024-03-11T21:47:46.577 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:46 smithi122 ceph-mon[27585]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T21:47:46.577 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:46 smithi122 ceph-mon[27585]: Cluster is now healthy 2024-03-11T21:47:46.577 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:46 smithi122 ceph-mon[27585]: overall HEALTH_OK 2024-03-11T21:47:46.577 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:46 smithi122 ceph-mon[27585]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:46.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:46 smithi165 ceph-mon[27942]: mon.b calling monitor election 2024-03-11T21:47:46.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:46 smithi165 ceph-mon[27942]: mon.b calling monitor election 2024-03-11T21:47:46.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:46 smithi165 ceph-mon[27942]: mon.c calling monitor election 2024-03-11T21:47:46.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:46 smithi165 ceph-mon[27942]: mon.a calling monitor election 2024-03-11T21:47:46.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:46 smithi165 ceph-mon[27942]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T21:47:46.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:46 smithi165 ceph-mon[27942]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:47:46.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:46 smithi165 ceph-mon[27942]: fsmap 2024-03-11T21:47:46.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:46 smithi165 ceph-mon[27942]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:47:46.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:46 smithi165 ceph-mon[27942]: mgrmap e13: y(active, since 63s) 2024-03-11T21:47:46.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:46 smithi165 ceph-mon[27942]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T21:47:46.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:46 smithi165 ceph-mon[27942]: Cluster is now healthy 2024-03-11T21:47:46.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:46 smithi165 ceph-mon[27942]: overall HEALTH_OK 2024-03-11T21:47:46.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:46 smithi165 ceph-mon[27942]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:46.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:46 smithi002 ceph-mon[25764]: mon.b calling monitor election 2024-03-11T21:47:46.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:46 smithi002 ceph-mon[25764]: mon.b calling monitor election 2024-03-11T21:47:46.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:46 smithi002 ceph-mon[25764]: mon.c calling monitor election 2024-03-11T21:47:46.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:46 smithi002 ceph-mon[25764]: mon.a calling monitor election 2024-03-11T21:47:46.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:46 smithi002 ceph-mon[25764]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T21:47:46.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:46 smithi002 ceph-mon[25764]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:47:46.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:46 smithi002 ceph-mon[25764]: fsmap 2024-03-11T21:47:46.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:46 smithi002 ceph-mon[25764]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:47:46.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:46 smithi002 ceph-mon[25764]: mgrmap e13: y(active, since 63s) 2024-03-11T21:47:46.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:46 smithi002 ceph-mon[25764]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T21:47:46.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:46 smithi002 ceph-mon[25764]: Cluster is now healthy 2024-03-11T21:47:46.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:46 smithi002 ceph-mon[25764]: overall HEALTH_OK 2024-03-11T21:47:46.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:46 smithi002 ceph-mon[25764]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:47.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:47 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:47.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:47 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:47 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:47 smithi165 ceph-mon[27942]: Reconfiguring mon.b (monmap changed)... 2024-03-11T21:47:47.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:47 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:47:47.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:47 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:47:47.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:47 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:47.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:47 smithi165 ceph-mon[27942]: Reconfiguring daemon mon.b on smithi122 2024-03-11T21:47:47.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:47 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:47 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:47 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:47 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.644 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:47:47 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:47:47.437+0000 7fade1d0d700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-11T21:47:47.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:47 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:47.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:47 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:47 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:47 smithi002 ceph-mon[25764]: Reconfiguring mon.b (monmap changed)... 2024-03-11T21:47:47.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:47 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:47:47.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:47 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:47:47.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:47 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:47.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:47 smithi002 ceph-mon[25764]: Reconfiguring daemon mon.b on smithi122 2024-03-11T21:47:47.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:47 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:47 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:47 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:47 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:47 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:47:47.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:47 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:47 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:47 smithi122 ceph-mon[27585]: Reconfiguring mon.b (monmap changed)... 2024-03-11T21:47:47.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:47 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:47:47.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:47 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:47:47.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:47 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:47.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:47 smithi122 ceph-mon[27585]: Reconfiguring daemon mon.b on smithi122 2024-03-11T21:47:47.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:47 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:47 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:47 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:47 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:47.735 INFO:teuthology.orchestra.run.smithi002.stdout:# minimal ceph.conf for 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:47:47.735 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2024-03-11T21:47:47.735 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:47:47.735 INFO:teuthology.orchestra.run.smithi002.stdout: mon_host = [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-03-11T21:47:48.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:48 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:48.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:48 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:48.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:48 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:48.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:48 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/2904569193' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:48.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:48 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:48.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:48 smithi002 ceph-mon[25764]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:48.357 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T21:47:48.357 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T21:47:48.357 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T21:47:48.388 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T21:47:48.388 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:47:48.457 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-11T21:47:48.457 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T21:47:48.488 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-11T21:47:48.488 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:47:48.554 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T21:47:48.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T21:47:48.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:48 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:48.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:48 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:48.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:48 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:48.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:48 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/2904569193' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:48.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:48 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:48.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:48 smithi165 ceph-mon[27942]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:48.592 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T21:47:48.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:47:48.663 INFO:tasks.cephadm:Adding mgr.y on smithi002 2024-03-11T21:47:48.663 INFO:tasks.cephadm:Adding mgr.x on smithi122 2024-03-11T21:47:48.663 INFO:tasks.cephadm:Adding mgr.z on smithi165 2024-03-11T21:47:48.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph orch apply mgr '3;smithi002=y;smithi122=x;smithi165=z' 2024-03-11T21:47:48.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:48 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:48.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:48 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:48.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:48 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:48.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:48 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/2904569193' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:48.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:48 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:48.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:48 smithi122 ceph-mon[27585]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:48.956 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:47:49.992 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:49 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:49.992 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:49 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:49.992 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:49 smithi165 ceph-mon[27942]: Reconfiguring mon.c (monmap changed)... 2024-03-11T21:47:49.992 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:49 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:47:49.992 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:49 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:47:49.992 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:49 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:49.992 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:49 smithi165 ceph-mon[27942]: Reconfiguring daemon mon.c on smithi165 2024-03-11T21:47:50.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:49 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:50.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:49 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:50.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:49 smithi002 ceph-mon[25764]: Reconfiguring mon.c (monmap changed)... 2024-03-11T21:47:50.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:49 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:47:50.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:49 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:47:50.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:49 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:50.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:49 smithi002 ceph-mon[25764]: Reconfiguring daemon mon.c on smithi165 2024-03-11T21:47:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:49 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:49 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:49 smithi122 ceph-mon[27585]: Reconfiguring mon.c (monmap changed)... 2024-03-11T21:47:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:49 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:47:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:49 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:47:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:49 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:49 smithi122 ceph-mon[27585]: Reconfiguring daemon mon.c on smithi165 2024-03-11T21:47:50.837 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled mgr update... 2024-03-11T21:47:50.850 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:50 smithi165 ceph-mon[27942]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:51.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:50 smithi002 ceph-mon[25764]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:51.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:50 smithi122 ceph-mon[27585]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:51.815 DEBUG:teuthology.orchestra.run.smithi122:mgr.x> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T21:47:51.818 DEBUG:teuthology.orchestra.run.smithi165:mgr.z> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@mgr.z.service 2024-03-11T21:47:51.820 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T21:47:51.820 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T21:47:51.820 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:47:51.837 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:47:51.837 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/nvme1n1 2024-03-11T21:47:51.892 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/nvme1n1 2024-03-11T21:47:51.892 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:47:51.893 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-11T21:47:51.893 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:47:51.893 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:47:51.893 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-03-11 21:47:33.841998266 +0000 2024-03-11T21:47:51.893 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-03-11 21:44:17.380378857 +0000 2024-03-11T21:47:51.893 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-03-11 21:44:17.380378857 +0000 2024-03-11T21:47:51.893 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-03-11 21:44:17.370379080 +0000 2024-03-11T21:47:51.893 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T21:47:51.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:51 smithi002 ceph-mon[25764]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=y;smithi122=x;smithi165=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:47:51.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:51 smithi002 ceph-mon[25764]: Saving service mgr spec with placement smithi002=y;smithi122=x;smithi165=z;count:3 2024-03-11T21:47:51.930 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:51 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:51.930 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:51 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:51.930 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:51 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:51.930 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:51 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:51.930 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:51 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:51.930 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:51 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:51.930 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:51 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:51.930 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:51 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:47:51.930 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:51 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:47:51.930 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:51 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:47:51.930 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:51 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:51.959 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-03-11T21:47:51.959 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-03-11T21:47:51.959 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000116272 s, 4.4 MB/s 2024-03-11T21:47:51.961 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T21:47:52.017 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/nvme2n1 2024-03-11T21:47:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/nvme2n1 2024-03-11T21:47:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:47:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-11T21:47:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:47:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:47:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-03-11 21:47:33.850998065 +0000 2024-03-11T21:47:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-03-11 21:44:17.531375483 +0000 2024-03-11T21:47:52.073 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-03-11 21:44:17.531375483 +0000 2024-03-11T21:47:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-03-11 21:44:17.525375617 +0000 2024-03-11T21:47:52.073 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T21:47:52.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:51 smithi165 ceph-mon[27942]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=y;smithi122=x;smithi165=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:47:52.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:51 smithi165 ceph-mon[27942]: Saving service mgr spec with placement smithi002=y;smithi122=x;smithi165=z;count:3 2024-03-11T21:47:52.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:51 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:52.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:51 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:52.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:51 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:52.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:51 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:52.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:51 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:52.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:51 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:52.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:51 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:52.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:51 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:47:52.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:51 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:47:52.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:51 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:47:52.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:51 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:52.138 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-03-11T21:47:52.138 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-03-11T21:47:52.138 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000110111 s, 4.6 MB/s 2024-03-11T21:47:52.139 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T21:47:52.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:51 smithi122 ceph-mon[27585]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=y;smithi122=x;smithi165=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:47:52.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:51 smithi122 ceph-mon[27585]: Saving service mgr spec with placement smithi002=y;smithi122=x;smithi165=z;count:3 2024-03-11T21:47:52.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:51 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:52.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:51 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:52.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:51 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:52.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:51 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:52.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:51 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:52.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:51 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:52.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:51 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:52.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:51 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:47:52.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:51 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:47:52.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:51 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:47:52.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:51 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:52.194 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/nvme3n1 2024-03-11T21:47:52.250 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/nvme3n1 2024-03-11T21:47:52.251 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:47:52.251 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-03-11T21:47:52.251 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:47:52.251 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:47:52.251 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-03-11 21:47:33.858997887 +0000 2024-03-11T21:47:52.251 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-03-11 21:44:17.685372043 +0000 2024-03-11T21:47:52.251 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-03-11 21:44:17.685372043 +0000 2024-03-11T21:47:52.251 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-03-11 21:44:17.675372267 +0000 2024-03-11T21:47:52.251 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T21:47:52.316 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-03-11T21:47:52.316 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-03-11T21:47:52.316 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000110712 s, 4.6 MB/s 2024-03-11T21:47:52.317 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T21:47:52.372 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/nvme4n1 2024-03-11T21:47:52.426 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/nvme4n1 2024-03-11T21:47:52.426 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:47:52.427 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T21:47:52.427 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:47:52.427 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:47:52.427 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-03-11 21:47:33.866997709 +0000 2024-03-11T21:47:52.427 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-03-11 21:44:17.838368626 +0000 2024-03-11T21:47:52.427 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-03-11 21:44:17.838368626 +0000 2024-03-11T21:47:52.427 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-03-11 21:44:17.832368760 +0000 2024-03-11T21:47:52.427 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T21:47:52.492 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-03-11T21:47:52.492 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-03-11T21:47:52.492 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000103721 s, 4.9 MB/s 2024-03-11T21:47:52.493 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T21:47:52.547 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-11T21:47:52.547 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:47:52.566 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:47:52.566 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/nvme1n1 2024-03-11T21:47:52.627 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/nvme1n1 2024-03-11T21:47:52.627 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:47:52.627 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-11T21:47:52.627 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:47:52.628 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:47:52.628 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-11 21:47:46.427342892 +0000 2024-03-11T21:47:52.628 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-11 21:44:19.045307826 +0000 2024-03-11T21:47:52.628 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-11 21:44:19.045307826 +0000 2024-03-11T21:47:52.628 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-11 21:44:19.034308090 +0000 2024-03-11T21:47:52.628 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T21:47:52.698 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-11T21:47:52.698 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-11T21:47:52.698 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000304642 s, 1.7 MB/s 2024-03-11T21:47:52.700 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T21:47:52.759 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/nvme2n1 2024-03-11T21:47:52.818 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/nvme2n1 2024-03-11T21:47:52.818 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:47:52.818 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-11T21:47:52.818 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:47:52.818 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:47:52.818 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-11 21:47:46.438342629 +0000 2024-03-11T21:47:52.818 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-11 21:44:19.192304296 +0000 2024-03-11T21:47:52.818 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-11 21:44:19.192304296 +0000 2024-03-11T21:47:52.818 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-11 21:44:19.181304560 +0000 2024-03-11T21:47:52.819 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T21:47:52.888 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-11T21:47:52.888 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-11T21:47:52.888 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000270799 s, 1.9 MB/s 2024-03-11T21:47:52.889 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T21:47:52.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:52 smithi002 ceph-mon[25764]: Deploying daemon mgr.z on smithi165 2024-03-11T21:47:52.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:52 smithi002 ceph-mon[25764]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:52.949 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/nvme3n1 2024-03-11T21:47:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/nvme3n1 2024-03-11T21:47:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:47:53.007 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-03-11T21:47:53.007 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:47:53.007 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:47:53.007 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-11 21:47:46.448342390 +0000 2024-03-11T21:47:53.007 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-11 21:44:19.340300741 +0000 2024-03-11T21:47:53.007 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-11 21:44:19.340300741 +0000 2024-03-11T21:47:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-11 21:44:19.328301029 +0000 2024-03-11T21:47:53.008 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T21:47:53.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:52 smithi165 ceph-mon[27942]: Deploying daemon mgr.z on smithi165 2024-03-11T21:47:53.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:52 smithi165 ceph-mon[27942]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:53.074 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:52 smithi122 ceph-mon[27585]: Deploying daemon mgr.z on smithi165 2024-03-11T21:47:53.074 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:52 smithi122 ceph-mon[27585]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:53.077 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-11T21:47:53.077 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-11T21:47:53.077 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000284584 s, 1.8 MB/s 2024-03-11T21:47:53.078 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T21:47:53.138 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/nvme4n1 2024-03-11T21:47:53.195 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/nvme4n1 2024-03-11T21:47:53.196 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:47:53.196 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-03-11T21:47:53.196 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:47:53.196 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:47:53.196 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-11 21:47:46.458342152 +0000 2024-03-11T21:47:53.196 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-11 21:44:19.488297187 +0000 2024-03-11T21:47:53.196 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-11 21:44:19.488297187 +0000 2024-03-11T21:47:53.196 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-11 21:44:19.477297451 +0000 2024-03-11T21:47:53.196 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T21:47:53.266 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-11T21:47:53.266 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-11T21:47:53.266 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000241208 s, 2.1 MB/s 2024-03-11T21:47:53.267 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T21:47:53.314 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:53 smithi165 systemd[1]: Starting Ceph mgr.z for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:47:53.327 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T21:47:53.327 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:47:53.344 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T21:47:53.344 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/nvme1n1 2024-03-11T21:47:53.399 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/nvme1n1 2024-03-11T21:47:53.399 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:47:53.399 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 764 Links: 1 Device type: 103,2 2024-03-11T21:47:53.399 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:47:53.399 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:47:53.399 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-11 21:47:46.287257823 +0000 2024-03-11T21:47:53.400 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-11 21:44:20.733441548 +0000 2024-03-11T21:47:53.400 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-11 21:44:20.733441548 +0000 2024-03-11T21:47:53.400 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-11 21:44:20.724441696 +0000 2024-03-11T21:47:53.400 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T21:47:53.467 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-11T21:47:53.467 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-11T21:47:53.467 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000204302 s, 2.5 MB/s 2024-03-11T21:47:53.468 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T21:47:53.524 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/nvme2n1 2024-03-11T21:47:53.580 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/nvme2n1 2024-03-11T21:47:53.580 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:47:53.580 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-11T21:47:53.580 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:47:53.580 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:47:53.581 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-11 21:47:46.297257672 +0000 2024-03-11T21:47:53.581 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-11 21:44:20.876439198 +0000 2024-03-11T21:47:53.581 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-11 21:44:20.876439198 +0000 2024-03-11T21:47:53.581 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-11 21:44:20.872439264 +0000 2024-03-11T21:47:53.581 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T21:47:53.582 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:53 smithi165 podman[30239]: 2024-03-11 21:47:53.384495289 +0000 UTC m=+0.017564851 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:47:53.582 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:53 smithi165 podman[30239]: 2024-03-11 21:47:53.505922909 +0000 UTC m=+0.138992541 container create 528f350cb9e22c624add3dcebfc8c12f0fea2bf1303f50b890bf0dce0c5dc84d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397) 2024-03-11T21:47:53.609 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-11T21:47:53.609 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-11T21:47:53.609 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000232606 s, 2.2 MB/s 2024-03-11T21:47:53.611 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T21:47:53.667 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/nvme3n1 2024-03-11T21:47:53.721 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/nvme3n1 2024-03-11T21:47:53.722 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:47:53.722 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-11T21:47:53.722 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:47:53.722 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:47:53.722 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-11 21:47:46.307257521 +0000 2024-03-11T21:47:53.722 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-11 21:44:21.045436420 +0000 2024-03-11T21:47:53.722 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-11 21:44:21.045436420 +0000 2024-03-11T21:47:53.722 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-11 21:44:21.036436568 +0000 2024-03-11T21:47:53.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T21:47:53.789 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-11T21:47:53.789 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-11T21:47:53.790 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000190107 s, 2.7 MB/s 2024-03-11T21:47:53.790 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T21:47:53.847 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/nvme4n1 2024-03-11T21:47:53.857 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:53 smithi165 podman[30239]: 2024-03-11 21:47:53.750251595 +0000 UTC m=+0.383321155 container init 528f350cb9e22c624add3dcebfc8c12f0fea2bf1303f50b890bf0dce0c5dc84d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T21:47:53.857 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:53 smithi165 podman[30239]: 2024-03-11 21:47:53.756354149 +0000 UTC m=+0.389423702 container start 528f350cb9e22c624add3dcebfc8c12f0fea2bf1303f50b890bf0dce0c5dc84d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-11T21:47:53.857 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:53 smithi165 ceph-mgr[30399]: -- 172.21.15.165:0/3916173137 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559bd443c1a0 con 0x559bd4413000 2024-03-11T21:47:53.858 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:53 smithi165 bash[30239]: 528f350cb9e22c624add3dcebfc8c12f0fea2bf1303f50b890bf0dce0c5dc84d 2024-03-11T21:47:53.872 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/nvme4n1 2024-03-11T21:47:53.872 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T21:47:53.872 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T21:47:53.872 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T21:47:53.873 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T21:47:53.873 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-11 21:47:46.318257355 +0000 2024-03-11T21:47:53.873 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-11 21:44:21.213433658 +0000 2024-03-11T21:47:53.873 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-11 21:44:21.213433658 +0000 2024-03-11T21:47:53.873 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-11 21:44:21.205433790 +0000 2024-03-11T21:47:53.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T21:47:53.939 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-11T21:47:53.939 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-11T21:47:53.939 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000219382 s, 2.3 MB/s 2024-03-11T21:47:53.940 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T21:47:53.997 INFO:tasks.cephadm:Deploying osd.0 on smithi002 with /dev/nvme4n1... 2024-03-11T21:47:53.998 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T21:47:54.271 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:54.313 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:53 smithi165 systemd[1]: Started Ceph mgr.z for 85326acc-dff0-11ee-95c8-87774f69a715. 2024-03-11T21:47:54.582 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:54 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:54.321+0000 7f003fc5e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:47:54.582 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:54 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:54.414+0000 7f003fc5e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:47:55.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:54 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:54 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:54 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:54 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:47:55.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:54 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:47:55.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:54 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:47:55.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:54 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:55.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:54 smithi002 ceph-mon[25764]: Deploying daemon mgr.x on smithi122 2024-03-11T21:47:55.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:54 smithi002 ceph-mon[25764]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:55.070 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:54 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.070 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:54 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.070 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:54 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:54 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:54 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:54 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:54 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:47:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:54 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:47:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:54 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:47:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:54 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:54 smithi122 ceph-mon[27585]: Deploying daemon mgr.x on smithi122 2024-03-11T21:47:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:54 smithi122 ceph-mon[27585]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:54 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:54 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:54 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.274 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:55 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:55.051+0000 7f003fc5e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:47:55.274 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:55 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:55.142+0000 7f003fc5e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:47:55.274 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:55 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:55.283+0000 7f003fc5e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:47:55.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:54 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:54 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:54 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:54 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:47:55.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:54 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:47:55.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:54 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:47:55.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:54 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:55.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:54 smithi165 ceph-mon[27942]: Deploying daemon mgr.x on smithi122 2024-03-11T21:47:55.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:54 smithi165 ceph-mon[27942]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:55.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:54 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:54 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:54 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:55.582 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:55 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:55.377+0000 7f003fc5e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:47:55.582 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:55 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:55.459+0000 7f003fc5e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:47:55.993 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:55 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:55.719+0000 7f003fc5e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:47:55.994 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:55 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:55.826+0000 7f003fc5e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:47:55.994 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:55 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:55.910+0000 7f003fc5e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:47:55.994 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:56 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:56.002+0000 7f003fc5e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:47:56.092 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T21:47:56.092 INFO:teuthology.orchestra.run.smithi002.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:47:56.092 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T21:47:56.092 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2024-03-11T21:47:56.092 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2024-03-11T21:47:56.092 INFO:teuthology.orchestra.run.smithi002.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00988914 s, 1.1 GB/s 2024-03-11T21:47:56.093 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2024-03-11T21:47:56.143 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:55 smithi122 systemd[1]: Starting Ceph mgr.x for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:47:56.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:56 smithi165 ceph-mon[27942]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:56.332 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:56 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:56.179+0000 7f003fc5e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:47:56.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:56 smithi002 ceph-mon[25764]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:56.408 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:56 smithi122 ceph-mon[27585]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:56.408 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:56 smithi122 podman[29228]: 2024-03-11 21:47:56.207452824 +0000 UTC m=+0.113179265 container create 31ff6547d310e16cd8ab69f9757d3dbd86cf5453e600cd8222ba885d6266bb35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-11T21:47:56.408 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:56 smithi122 podman[29228]: 2024-03-11 21:47:56.111717177 +0000 UTC m=+0.017443618 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:47:56.408 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:56 smithi122 podman[29228]: 2024-03-11 21:47:56.417839705 +0000 UTC m=+0.323566162 container init 31ff6547d310e16cd8ab69f9757d3dbd86cf5453e600cd8222ba885d6266bb35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-11T21:47:56.673 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:56 smithi122 podman[29228]: 2024-03-11 21:47:56.423258926 +0000 UTC m=+0.328985370 container start 31ff6547d310e16cd8ab69f9757d3dbd86cf5453e600cd8222ba885d6266bb35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0) 2024-03-11T21:47:56.673 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:56 smithi122 ceph-mgr[29267]: -- 172.21.15.122:0/83428203 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e12403a1a0 con 0x55e124010c00 2024-03-11T21:47:56.674 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:56 smithi122 bash[29228]: 31ff6547d310e16cd8ab69f9757d3dbd86cf5453e600cd8222ba885d6266bb35 2024-03-11T21:47:56.674 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:56 smithi122 systemd[1]: Started Ceph mgr.x for 85326acc-dff0-11ee-95c8-87774f69a715. 2024-03-11T21:47:56.736 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi002:/dev/nvme4n1 2024-03-11T21:47:56.980 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:57.423 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:47:57.000+0000 7f5d6b57c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:47:57.424 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:47:57.095+0000 7f5d6b57c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:47:57.428 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:57.178+0000 7f003fc5e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:47:57.429 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:57.312+0000 7f003fc5e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:47:57.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:57 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:57 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:57 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:57 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:57 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:57.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:57 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:57.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:57 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:57.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:57 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:57 smithi002 ceph-mon[25764]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:47:57.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:57 smithi002 ceph-mon[25764]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:47:57.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:57 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:47:57.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:57 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:47:57.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:57 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:57.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:57 smithi002 ceph-mon[25764]: Reconfiguring daemon mgr.y on smithi002 2024-03-11T21:47:57.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:57 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.689 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:57.661+0000 7f003fc5e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:47:57.689 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.689 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.689 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.689 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.689 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:57.689 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:57.690 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:57.690 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.690 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-mon[27942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:47:57.690 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-mon[27942]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:47:57.690 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:47:57.690 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:47:57.690 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:57.690 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-mon[27942]: Reconfiguring daemon mgr.y on smithi002 2024-03-11T21:47:57.690 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.736 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.736 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.736 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.736 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.736 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:57.736 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:57.736 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:57.737 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.737 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-mon[27585]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:47:57.737 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-mon[27585]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:47:57.737 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:47:57.737 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:47:57.737 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:57.737 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-mon[27585]: Reconfiguring daemon mgr.y on smithi002 2024-03-11T21:47:57.737 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:57.947 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:57.751+0000 7f003fc5e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:47:57.947 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:57 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:57.945+0000 7f003fc5e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:47:58.060 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:47:57.744+0000 7f5d6b57c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:47:58.060 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:47:57.836+0000 7f5d6b57c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:47:58.060 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:57 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:47:57.974+0000 7f5d6b57c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:47:58.250 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:58 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:58.145+0000 7f003fc5e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:47:58.412 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:58 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:47:58.068+0000 7f5d6b57c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:47:58.412 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:58 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:47:58.150+0000 7f5d6b57c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:47:58.582 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:58 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:58.456+0000 7f003fc5e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:47:58.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:58 smithi122 ceph-mon[27585]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:58.673 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:58 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:47:58.429+0000 7f5d6b57c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:47:58.674 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:58 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:47:58.541+0000 7f5d6b57c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:47:58.674 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:58 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:47:58.630+0000 7f5d6b57c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:47:58.837 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:58 smithi002 ceph-mon[25764]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:58.931 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:58 smithi165 ceph-mon[27942]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:47:58.931 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:58 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:58.735+0000 7f003fc5e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:47:58.989 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:58 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:47:58.725+0000 7f5d6b57c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:47:58.989 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:58 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:47:58.903+0000 7f5d6b57c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:47:59.200 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:58 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:58.994+0000 7f003fc5e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:47:59.799 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:59 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:47:59.800 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:59 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:47:59.800 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:59 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:59.800 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:59 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:59.800 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:59 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:59.800 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:47:59 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:59.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:59 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:47:59.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:59 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:47:59.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:59 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:59.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:59 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:59.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:59 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:59.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:47:59 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:59.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:59 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:47:59.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:59 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:47:59.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:59 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:59.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:59 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:59.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:59 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:47:59.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:47:59 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:00.082 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:47:59 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:47:59.819+0000 7f003fc5e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:48:00.253 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:47:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:47:59.969+0000 7f5d6b57c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:48:00.253 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:48:00.129+0000 7f5d6b57c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:48:00.553 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:48:00.474+0000 7f5d6b57c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:48:00.553 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:48:00.562+0000 7f5d6b57c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:48:00.657 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:48:00 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:48:00.394+0000 7f003fc5e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:48:00.657 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:48:00 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:48:00.489+0000 7f003fc5e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:48:00.923 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:48:00.757+0000 7f5d6b57c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:48:00.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-mon[27585]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:48:00.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:00.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:00.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:00.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:00.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:00.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:00.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-mon[27585]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:48:00.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:00.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:00.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:00.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-mon[27585]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:00.964 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:00 smithi002 ceph-mon[25764]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:48:00.964 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:00 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:00.964 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:00 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:00.964 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:00 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:00.964 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:00 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:00.964 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:00 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:00.964 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:00 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:00.965 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:00 smithi002 ceph-mon[25764]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:48:00.965 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:00 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:00.965 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:00 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:00.965 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:00 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:00.965 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:00 smithi002 ceph-mon[25764]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:01.082 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:48:00 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:48:00.700+0000 7f003fc5e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:48:01.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:00 smithi165 ceph-mon[27942]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:48:01.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:00 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:01.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:00 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:01.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:00 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:01.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:00 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:01.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:00 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:01.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:00 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:01.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:00 smithi165 ceph-mon[27942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:48:01.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:00 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:01.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:00 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:01.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:00 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:01.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:00 smithi165 ceph-mon[27942]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:01.252 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:48:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:48:00.961+0000 7f5d6b57c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:48:01.515 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:48:01.260+0000 7f5d6b57c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:48:01.515 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:48:01.523+0000 7f5d6b57c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:48:01.923 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:48:01.754+0000 7f5d6b57c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:48:01.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-mon[27585]: Standby manager daemon z started 2024-03-11T21:48:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.165:0/1476616270' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:48:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.165:0/1476616270' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:48:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.165:0/1476616270' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:48:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.165:0/1476616270' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:48:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:01.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:01 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:02.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:01 smithi165 ceph-mon[27942]: Standby manager daemon z started 2024-03-11T21:48:02.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:01 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.165:0/1476616270' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:48:02.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:01 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.165:0/1476616270' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:48:02.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:01 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.165:0/1476616270' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:48:02.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:01 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.165:0/1476616270' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:48:02.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:01 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:02.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:01 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:02.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:01 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:02.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:01 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:02.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:01 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:02.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:01 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:02.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:01 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:02.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:01 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:02.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:01 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:02.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:01 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:02.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:01 smithi002 ceph-mon[25764]: Standby manager daemon z started 2024-03-11T21:48:02.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:01 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.165:0/1476616270' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:48:02.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:01 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.165:0/1476616270' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:48:02.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:01 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.165:0/1476616270' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:48:02.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:01 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.165:0/1476616270' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:48:02.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:01 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:02.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:01 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:02.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:01 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:02.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:01 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:02.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:01 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:02.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:01 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:02.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:01 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:02.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:01 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:02.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:01 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:02.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:01 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:02.923 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:48:02 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:48:02.537+0000 7f5d6b57c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:48:02.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:02 smithi122 ceph-mon[27585]: mgrmap e14: y(active, since 79s), standbys: z 2024-03-11T21:48:02.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:02 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:48:02.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:02 smithi122 ceph-mon[27585]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:02.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:02 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/1154263249' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac24d686-0fd0-411c-b9ea-6e2ea65d079d"}]: dispatch 2024-03-11T21:48:02.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:02 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/1154263249' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac24d686-0fd0-411c-b9ea-6e2ea65d079d"}]': finished 2024-03-11T21:48:02.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:02 smithi122 ceph-mon[27585]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:48:02.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:02 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:48:03.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:02 smithi165 ceph-mon[27942]: mgrmap e14: y(active, since 79s), standbys: z 2024-03-11T21:48:03.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:02 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:48:03.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:02 smithi165 ceph-mon[27942]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:03.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:02 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/1154263249' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac24d686-0fd0-411c-b9ea-6e2ea65d079d"}]: dispatch 2024-03-11T21:48:03.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:02 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/1154263249' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac24d686-0fd0-411c-b9ea-6e2ea65d079d"}]': finished 2024-03-11T21:48:03.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:02 smithi165 ceph-mon[27942]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:48:03.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:02 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:48:03.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:02 smithi002 ceph-mon[25764]: mgrmap e14: y(active, since 79s), standbys: z 2024-03-11T21:48:03.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:02 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:48:03.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:02 smithi002 ceph-mon[25764]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:03.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:02 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1154263249' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ac24d686-0fd0-411c-b9ea-6e2ea65d079d"}]: dispatch 2024-03-11T21:48:03.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:02 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1154263249' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ac24d686-0fd0-411c-b9ea-6e2ea65d079d"}]': finished 2024-03-11T21:48:03.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:02 smithi002 ceph-mon[25764]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:48:03.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:02 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:48:03.398 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:48:03 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:48:03.099+0000 7f5d6b57c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:48:03.398 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:48:03 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:48:03.195+0000 7f5d6b57c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:48:03.399 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:48:03 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:48:03.406+0000 7f5d6b57c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:48:03.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:03 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/2069634377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:48:03.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:03 smithi002 ceph-mon[25764]: Standby manager daemon x started 2024-03-11T21:48:03.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:03 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.122:0/1399975548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:48:03.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:03 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.122:0/1399975548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:48:03.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:03 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.122:0/1399975548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:48:03.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:03 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.122:0/1399975548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:48:04.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:03 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/2069634377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:48:04.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:03 smithi165 ceph-mon[27942]: Standby manager daemon x started 2024-03-11T21:48:04.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:03 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.122:0/1399975548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:48:04.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:03 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.122:0/1399975548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:48:04.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:03 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.122:0/1399975548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:48:04.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:03 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.122:0/1399975548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:48:04.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:03 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/2069634377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:48:04.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:03 smithi122 ceph-mon[27585]: Standby manager daemon x started 2024-03-11T21:48:04.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:03 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.122:0/1399975548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:48:04.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:03 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.122:0/1399975548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:48:04.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:03 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.122:0/1399975548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:48:04.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:03 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.122:0/1399975548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:48:05.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:04 smithi165 ceph-mon[27942]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-11T21:48:05.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:04 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:48:05.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:04 smithi165 ceph-mon[27942]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:05.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:04 smithi002 ceph-mon[25764]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-11T21:48:05.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:04 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:48:05.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:04 smithi002 ceph-mon[25764]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:05.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:04 smithi122 ceph-mon[27585]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-11T21:48:05.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:04 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:48:05.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:04 smithi122 ceph-mon[27585]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:06.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:06 smithi002 ceph-mon[25764]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:06.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:06 smithi122 ceph-mon[27585]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:06.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:06 smithi165 ceph-mon[27942]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:08.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:08 smithi002 ceph-mon[25764]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:08.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:08 smithi122 ceph-mon[27585]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:08.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:08 smithi165 ceph-mon[27942]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:09.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:09 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:48:09.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:09 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:09.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:09 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:48:09.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:09 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:09.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:09 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:48:09.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:09 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:10.529 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:10 smithi002 ceph-mon[25764]: Deploying daemon osd.0 on smithi002 2024-03-11T21:48:10.529 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:10 smithi002 ceph-mon[25764]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:10.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:10 smithi165 ceph-mon[27942]: Deploying daemon osd.0 on smithi002 2024-03-11T21:48:10.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:10 smithi165 ceph-mon[27942]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:10.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:10 smithi122 ceph-mon[27585]: Deploying daemon osd.0 on smithi002 2024-03-11T21:48:10.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:10 smithi122 ceph-mon[27585]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:12.335 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:12 smithi002 ceph-mon[25764]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:12.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:12 smithi122 ceph-mon[27585]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:12.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:12 smithi165 ceph-mon[27942]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:14.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:14 smithi002 ceph-mon[25764]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:14.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:14 smithi122 ceph-mon[27585]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:14.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:14 smithi165 ceph-mon[27942]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:16.006 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:15 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:16.006 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:15 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:16.006 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:15 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:16.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:15 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:16.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:15 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:16.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:15 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:16.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:15 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:16.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:15 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:16.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:15 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.130 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.130 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.130 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:17.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:17.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:17.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:16 smithi002 ceph-mon[25764]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:17.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:16 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:16 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:16 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:17.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:16 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:17.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:16 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:17.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:16 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:16 smithi122 ceph-mon[27585]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:17.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:16 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:16 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:16 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:16 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:16 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:16 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:17.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:16 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:17.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:16 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:17.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:16 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:16 smithi165 ceph-mon[27942]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:17.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:16 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:16 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:17.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:16 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:18.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:18 smithi165 ceph-mon[27942]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:18.327 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:18 smithi002 ceph-mon[25764]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:18.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:18 smithi122 ceph-mon[27585]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:19.650 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 0 on host 'smithi002' 2024-03-11T21:48:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:19 smithi002 ceph-mon[25764]: from='osd.0 [v2:172.21.15.2:6802/360299831,v1:172.21.15.2:6803/360299831]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:48:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:48:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:19.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:19.896 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:19 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:19 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:19 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:19 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:19 smithi122 ceph-mon[27585]: from='osd.0 [v2:172.21.15.2:6802/360299831,v1:172.21.15.2:6803/360299831]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:48:19.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:19 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:19 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:19 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:19 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:19.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:19 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:19 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:48:19.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:19 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:19 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:19.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:19 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:19.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:19 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:19.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:19 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:20.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:19 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:20.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:19 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:20.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:19 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:20.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:19 smithi165 ceph-mon[27942]: from='osd.0 [v2:172.21.15.2:6802/360299831,v1:172.21.15.2:6803/360299831]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:48:20.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:19 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:20.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:19 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:20.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:19 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:20.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:19 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:20.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:19 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:20.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:19 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:48:20.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:19 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:20.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:19 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:20.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:19 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:20.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:19 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:20.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:19 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:20.264 DEBUG:teuthology.orchestra.run.smithi002:osd.0> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T21:48:20.266 INFO:tasks.cephadm:Deploying osd.1 on smithi122 with /dev/nvme4n1... 2024-03-11T21:48:20.266 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T21:48:20.845 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:48:20.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:20 smithi122 ceph-mon[27585]: Adjusting osd_memory_target on smithi002 to 16069M 2024-03-11T21:48:20.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:20 smithi122 ceph-mon[27585]: from='osd.0 [v2:172.21.15.2:6802/360299831,v1:172.21.15.2:6803/360299831]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:48:20.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:20 smithi122 ceph-mon[27585]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:48:20.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:20 smithi122 ceph-mon[27585]: from='osd.0 [v2:172.21.15.2:6802/360299831,v1:172.21.15.2:6803/360299831]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T21:48:20.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:20 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:48:20.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:20 smithi122 ceph-mon[27585]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:21.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:20 smithi165 ceph-mon[27942]: Adjusting osd_memory_target on smithi002 to 16069M 2024-03-11T21:48:21.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:20 smithi165 ceph-mon[27942]: from='osd.0 [v2:172.21.15.2:6802/360299831,v1:172.21.15.2:6803/360299831]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:48:21.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:20 smithi165 ceph-mon[27942]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:48:21.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:20 smithi165 ceph-mon[27942]: from='osd.0 [v2:172.21.15.2:6802/360299831,v1:172.21.15.2:6803/360299831]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T21:48:21.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:20 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:48:21.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:20 smithi165 ceph-mon[27942]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:21.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:20 smithi002 ceph-mon[25764]: Adjusting osd_memory_target on smithi002 to 16069M 2024-03-11T21:48:21.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:20 smithi002 ceph-mon[25764]: from='osd.0 [v2:172.21.15.2:6802/360299831,v1:172.21.15.2:6803/360299831]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:48:21.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:20 smithi002 ceph-mon[25764]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:48:21.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:20 smithi002 ceph-mon[25764]: from='osd.0 [v2:172.21.15.2:6802/360299831,v1:172.21.15.2:6803/360299831]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T21:48:21.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:20 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:48:21.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:20 smithi002 ceph-mon[25764]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:21.145 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:48:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-0[38233]: 2024-03-11T21:48:20.679+0000 7fb883094700 -1 osd.0 0 waiting for initial osdmap 2024-03-11T21:48:21.146 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:48:20 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-0[38233]: 2024-03-11T21:48:20.684+0000 7fb87ee87700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:48:21.874 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:21 smithi122 ceph-mon[27585]: from='osd.0 [v2:172.21.15.2:6802/360299831,v1:172.21.15.2:6803/360299831]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-03-11T21:48:21.874 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:21 smithi122 ceph-mon[27585]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:48:21.874 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:48:21.875 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:48:22.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:21 smithi165 ceph-mon[27942]: from='osd.0 [v2:172.21.15.2:6802/360299831,v1:172.21.15.2:6803/360299831]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-03-11T21:48:22.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:21 smithi165 ceph-mon[27942]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:48:22.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:48:22.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:48:22.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:21 smithi002 ceph-mon[25764]: from='osd.0 [v2:172.21.15.2:6802/360299831,v1:172.21.15.2:6803/360299831]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-03-11T21:48:22.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:21 smithi002 ceph-mon[25764]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:48:22.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:48:22.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:48:22.832 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:22 smithi165 ceph-mon[27942]: purged_snaps scrub starts 2024-03-11T21:48:22.832 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:22 smithi165 ceph-mon[27942]: purged_snaps scrub ok 2024-03-11T21:48:22.832 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:22 smithi165 ceph-mon[27942]: osd.0 [v2:172.21.15.2:6802/360299831,v1:172.21.15.2:6803/360299831] boot 2024-03-11T21:48:22.832 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:22 smithi165 ceph-mon[27942]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:48:22.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:22 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:48:22.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:22 smithi165 ceph-mon[27942]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:22.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:22 smithi122 ceph-mon[27585]: purged_snaps scrub starts 2024-03-11T21:48:22.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:22 smithi122 ceph-mon[27585]: purged_snaps scrub ok 2024-03-11T21:48:22.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:22 smithi122 ceph-mon[27585]: osd.0 [v2:172.21.15.2:6802/360299831,v1:172.21.15.2:6803/360299831] boot 2024-03-11T21:48:22.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:22 smithi122 ceph-mon[27585]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:48:22.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:22 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:48:22.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:22 smithi122 ceph-mon[27585]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:23.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:22 smithi002 ceph-mon[25764]: purged_snaps scrub starts 2024-03-11T21:48:23.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:22 smithi002 ceph-mon[25764]: purged_snaps scrub ok 2024-03-11T21:48:23.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:22 smithi002 ceph-mon[25764]: osd.0 [v2:172.21.15.2:6802/360299831,v1:172.21.15.2:6803/360299831] boot 2024-03-11T21:48:23.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:22 smithi002 ceph-mon[25764]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:48:23.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:22 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:48:23.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:22 smithi002 ceph-mon[25764]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:48:23.225 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T21:48:23.225 INFO:teuthology.orchestra.run.smithi122.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:48:23.225 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T21:48:23.225 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-03-11T21:48:23.225 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-03-11T21:48:23.225 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012049 s, 870 MB/s 2024-03-11T21:48:23.225 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-03-11T21:48:23.800 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:23 smithi122 ceph-mon[27585]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:48:23.828 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi122:/dev/nvme4n1 2024-03-11T21:48:24.059 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:48:24.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:23 smithi165 ceph-mon[27942]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:48:24.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:23 smithi002 ceph-mon[25764]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:48:24.831 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:24 smithi122 ceph-mon[27585]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:25.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:24 smithi165 ceph-mon[27942]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:25.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:24 smithi002 ceph-mon[25764]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:26.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:25 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:48:26.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:25 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:48:26.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:25 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:26.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:48:26.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:48:26.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:26.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:25 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:48:26.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:25 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:48:26.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:25 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:26.992 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:26 smithi122 ceph-mon[27585]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:48:26.992 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:26 smithi122 ceph-mon[27585]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:27.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:26 smithi165 ceph-mon[27942]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:48:27.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:26 smithi165 ceph-mon[27942]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:27.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:26 smithi002 ceph-mon[25764]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:48:27.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:26 smithi002 ceph-mon[25764]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:28.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:28 smithi122 ceph-mon[27585]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:28.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:28 smithi165 ceph-mon[27942]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:28.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:28 smithi002 ceph-mon[25764]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:29.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:29 smithi122 ceph-mon[27585]: from='client.? 172.21.15.122:0/1228403569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "126d0d98-285a-414f-a848-fdc47e6e3516"}]: dispatch 2024-03-11T21:48:29.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:29 smithi122 ceph-mon[27585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "126d0d98-285a-414f-a848-fdc47e6e3516"}]: dispatch 2024-03-11T21:48:29.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:29 smithi122 ceph-mon[27585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "126d0d98-285a-414f-a848-fdc47e6e3516"}]': finished 2024-03-11T21:48:29.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:29 smithi122 ceph-mon[27585]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:48:29.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:29 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:29.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:29 smithi165 ceph-mon[27942]: from='client.? 172.21.15.122:0/1228403569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "126d0d98-285a-414f-a848-fdc47e6e3516"}]: dispatch 2024-03-11T21:48:29.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:29 smithi165 ceph-mon[27942]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "126d0d98-285a-414f-a848-fdc47e6e3516"}]: dispatch 2024-03-11T21:48:29.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:29 smithi165 ceph-mon[27942]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "126d0d98-285a-414f-a848-fdc47e6e3516"}]': finished 2024-03-11T21:48:29.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:29 smithi165 ceph-mon[27942]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:48:29.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:29 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:29.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:29 smithi002 ceph-mon[25764]: from='client.? 172.21.15.122:0/1228403569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "126d0d98-285a-414f-a848-fdc47e6e3516"}]: dispatch 2024-03-11T21:48:29.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:29 smithi002 ceph-mon[25764]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "126d0d98-285a-414f-a848-fdc47e6e3516"}]: dispatch 2024-03-11T21:48:29.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:29 smithi002 ceph-mon[25764]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "126d0d98-285a-414f-a848-fdc47e6e3516"}]': finished 2024-03-11T21:48:29.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:29 smithi002 ceph-mon[25764]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:48:29.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:29 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:30.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:30 smithi122 ceph-mon[27585]: from='client.? 172.21.15.122:0/926072241' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:48:30.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:30 smithi122 ceph-mon[27585]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:30.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:30 smithi165 ceph-mon[27942]: from='client.? 172.21.15.122:0/926072241' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:48:30.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:30 smithi165 ceph-mon[27942]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:30.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:30 smithi002 ceph-mon[25764]: from='client.? 172.21.15.122:0/926072241' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:48:30.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:30 smithi002 ceph-mon[25764]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:32.368 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:32 smithi122 ceph-mon[27585]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:32.447 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:32 smithi002 ceph-mon[25764]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:32.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:32 smithi165 ceph-mon[27942]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:34.257 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:34 smithi122 ceph-mon[27585]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:34.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:34 smithi165 ceph-mon[27942]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:34.631 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:34 smithi002 ceph-mon[25764]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:35.840 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:35 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:48:35.840 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:35 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:35.840 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:35 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:35.840 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:35 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:35.840 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:35 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:35.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:35 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:48:35.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:35 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:35.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:35 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:35.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:35 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:35.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:35 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:36.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:35 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:48:36.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:35 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:36.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:35 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:36.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:35 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:36.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:35 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:36.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:36 smithi122 ceph-mon[27585]: Deploying daemon osd.1 on smithi122 2024-03-11T21:48:36.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:36 smithi122 ceph-mon[27585]: Detected new or changed devices on smithi002 2024-03-11T21:48:36.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:36 smithi122 ceph-mon[27585]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:36 smithi002 ceph-mon[25764]: Deploying daemon osd.1 on smithi122 2024-03-11T21:48:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:36 smithi002 ceph-mon[25764]: Detected new or changed devices on smithi002 2024-03-11T21:48:36.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:36 smithi002 ceph-mon[25764]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:37.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:36 smithi165 ceph-mon[27942]: Deploying daemon osd.1 on smithi122 2024-03-11T21:48:37.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:36 smithi165 ceph-mon[27942]: Detected new or changed devices on smithi002 2024-03-11T21:48:37.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:36 smithi165 ceph-mon[27942]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:38.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:38 smithi122 ceph-mon[27585]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:38.499 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:38 smithi165 ceph-mon[27942]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:38.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:38 smithi002 ceph-mon[25764]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:39.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:39 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:39.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:39 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:39.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:39 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:39.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:39 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:39.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:39 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:39.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:39 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:40.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:39 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:40.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:39 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:40.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:39 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:40.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:39 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:40.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:39 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:40.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:39 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:40.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:39 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:40.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:39 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:40.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:39 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:40.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:39 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:40.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:39 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:40.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:39 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:41.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:40 smithi122 ceph-mon[27585]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:41.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:40 smithi165 ceph-mon[27942]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:41.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:40 smithi002 ceph-mon[25764]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:42.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:42 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:42.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:42 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:42.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:42 smithi122 ceph-mon[27585]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:42.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:42 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:42.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:42 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:42.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:42 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:42.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:42 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:42.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:42 smithi122 ceph-mon[27585]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:42.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:42 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:42.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:42 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:48:42.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:42 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:42.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:42 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:42.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:42 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:43.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:43.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:43.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:42 smithi002 ceph-mon[25764]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:43.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:43.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:43.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:43.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:43.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:42 smithi002 ceph-mon[25764]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:43.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:43.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:48:43.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:43.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:43.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:42 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:43.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:42 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:42 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:42 smithi165 ceph-mon[27942]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:42 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:42 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:42 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:42 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:42 smithi165 ceph-mon[27942]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:42 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:42 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:48:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:42 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:42 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:42 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:43 smithi002 ceph-mon[25764]: Reconfiguring osd.1 (unknown last config time)... 2024-03-11T21:48:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:43 smithi002 ceph-mon[25764]: Reconfiguring daemon osd.1 on smithi122 2024-03-11T21:48:44.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:43 smithi002 ceph-mon[25764]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T21:48:44.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:43 smithi122 ceph-mon[27585]: Reconfiguring osd.1 (unknown last config time)... 2024-03-11T21:48:44.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:43 smithi122 ceph-mon[27585]: Reconfiguring daemon osd.1 on smithi122 2024-03-11T21:48:44.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:43 smithi122 ceph-mon[27585]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T21:48:44.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:43 smithi165 ceph-mon[27942]: Reconfiguring osd.1 (unknown last config time)... 2024-03-11T21:48:44.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:43 smithi165 ceph-mon[27942]: Reconfiguring daemon osd.1 on smithi122 2024-03-11T21:48:44.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:43 smithi165 ceph-mon[27942]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T21:48:44.923 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:44 smithi122 ceph-mon[27585]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:45.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:44 smithi002 ceph-mon[25764]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:45.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:44 smithi165 ceph-mon[27942]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:46.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:45 smithi002 ceph-mon[25764]: from='osd.1 [v2:172.21.15.122:6800/377314645,v1:172.21.15.122:6801/377314645]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:48:46.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:45 smithi002 ceph-mon[25764]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:48:46.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:45 smithi122 ceph-mon[27585]: from='osd.1 [v2:172.21.15.122:6800/377314645,v1:172.21.15.122:6801/377314645]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:48:46.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:45 smithi122 ceph-mon[27585]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:48:46.239 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 1 on host 'smithi122' 2024-03-11T21:48:46.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:45 smithi165 ceph-mon[27942]: from='osd.1 [v2:172.21.15.122:6800/377314645,v1:172.21.15.122:6801/377314645]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:48:46.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:45 smithi165 ceph-mon[27942]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:48:47.107 DEBUG:teuthology.orchestra.run.smithi122:osd.1> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T21:48:47.110 INFO:tasks.cephadm:Deploying osd.2 on smithi165 with /dev/nvme4n1... 2024-03-11T21:48:47.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T21:48:47.135 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:46 smithi122 ceph-mon[27585]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:48:47.135 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:46 smithi122 ceph-mon[27585]: from='osd.1 [v2:172.21.15.122:6800/377314645,v1:172.21.15.122:6801/377314645]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-11T21:48:47.135 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:46 smithi122 ceph-mon[27585]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:48:47.135 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:46 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:47.135 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:46 smithi122 ceph-mon[27585]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-11T21:48:47.136 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:46 smithi122 ceph-mon[27585]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:47.136 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:46 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:47.136 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:46 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:47.140 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:46 smithi165 ceph-mon[27942]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:48:47.141 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:46 smithi165 ceph-mon[27942]: from='osd.1 [v2:172.21.15.122:6800/377314645,v1:172.21.15.122:6801/377314645]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-11T21:48:47.141 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:46 smithi165 ceph-mon[27942]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:48:47.141 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:46 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:47.141 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:46 smithi165 ceph-mon[27942]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-11T21:48:47.141 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:46 smithi165 ceph-mon[27942]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:47.141 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:46 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:47.141 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:46 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:47.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:46 smithi002 ceph-mon[25764]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:48:47.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:46 smithi002 ceph-mon[25764]: from='osd.1 [v2:172.21.15.122:6800/377314645,v1:172.21.15.122:6801/377314645]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-11T21:48:47.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:46 smithi002 ceph-mon[25764]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:48:47.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:46 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:47.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:46 smithi002 ceph-mon[25764]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-11T21:48:47.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:46 smithi002 ceph-mon[25764]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:47.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:46 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:47.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:46 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:47.359 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:48:47.424 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:48:47 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-1[33104]: 2024-03-11T21:48:47.327+0000 7f8041bbe700 -1 osd.1 0 waiting for initial osdmap 2024-03-11T21:48:47.424 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:48:47 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-1[33104]: 2024-03-11T21:48:47.329+0000 7f80399a9700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:48:47.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:47 smithi122 ceph-mon[27585]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-11T21:48:47.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:47 smithi122 ceph-mon[27585]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:48:47.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:47 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:47.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:47 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:47.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:47 smithi122 ceph-mon[27585]: from='osd.1 ' entity='osd.1' 2024-03-11T21:48:48.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:47 smithi165 ceph-mon[27942]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-11T21:48:48.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:47 smithi165 ceph-mon[27942]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:48:48.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:47 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:48.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:47 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:48.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:47 smithi165 ceph-mon[27942]: from='osd.1 ' entity='osd.1' 2024-03-11T21:48:48.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:47 smithi002 ceph-mon[25764]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-11T21:48:48.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:47 smithi002 ceph-mon[25764]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:48:48.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:47 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:48.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:47 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:48.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:47 smithi002 ceph-mon[25764]: from='osd.1 ' entity='osd.1' 2024-03-11T21:48:49.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:48 smithi165 ceph-mon[27942]: purged_snaps scrub starts 2024-03-11T21:48:49.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:48 smithi165 ceph-mon[27942]: purged_snaps scrub ok 2024-03-11T21:48:49.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:48 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:49.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:48 smithi165 ceph-mon[27942]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:49.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:48 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:49.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:48 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:49.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:48 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:49.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:48 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:48:49.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:48 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:49.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:48 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:49.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:48 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:49.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:48 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:49.084 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:48 smithi165 ceph-mon[27942]: osd.1 [v2:172.21.15.122:6800/377314645,v1:172.21.15.122:6801/377314645] boot 2024-03-11T21:48:49.084 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:48 smithi165 ceph-mon[27942]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:48:49.084 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:48 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:49.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:48 smithi002 ceph-mon[25764]: purged_snaps scrub starts 2024-03-11T21:48:49.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:48 smithi002 ceph-mon[25764]: purged_snaps scrub ok 2024-03-11T21:48:49.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:48 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:49.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:48 smithi002 ceph-mon[25764]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:49.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:48 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:49.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:48 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:49.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:48 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:49.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:48 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:48:49.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:48 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:49.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:48 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:49.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:48 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:49.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:48 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:49.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:48 smithi002 ceph-mon[25764]: osd.1 [v2:172.21.15.122:6800/377314645,v1:172.21.15.122:6801/377314645] boot 2024-03-11T21:48:49.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:48 smithi002 ceph-mon[25764]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:48:49.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:48 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:49.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:48 smithi122 ceph-mon[27585]: purged_snaps scrub starts 2024-03-11T21:48:49.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:48 smithi122 ceph-mon[27585]: purged_snaps scrub ok 2024-03-11T21:48:49.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:48 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:49.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:48 smithi122 ceph-mon[27585]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:49.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:48 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:49.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:48 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:49.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:48 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:49.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:48 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:48:49.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:48 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:49.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:48 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:49.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:48 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:49.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:48 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:49.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:48 smithi122 ceph-mon[27585]: osd.1 [v2:172.21.15.122:6800/377314645,v1:172.21.15.122:6801/377314645] boot 2024-03-11T21:48:49.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:48 smithi122 ceph-mon[27585]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:48:49.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:48 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:48:49.449 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T21:48:49.449 INFO:teuthology.orchestra.run.smithi165.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T21:48:49.449 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T21:48:49.449 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-03-11T21:48:49.449 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-03-11T21:48:49.449 INFO:teuthology.orchestra.run.smithi165.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124793 s, 840 MB/s 2024-03-11T21:48:49.449 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-03-11T21:48:50.025 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:49 smithi165 ceph-mon[27942]: Adjusting osd_memory_target on smithi122 to 16064M 2024-03-11T21:48:50.025 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:49 smithi165 ceph-mon[27942]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:50.025 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:49 smithi165 ceph-mon[27942]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-11T21:48:50.026 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:49 smithi165 ceph-mon[27942]: Cluster is now healthy 2024-03-11T21:48:50.026 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:49 smithi165 ceph-mon[27942]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:48:50.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi165:/dev/nvme4n1 2024-03-11T21:48:50.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:49 smithi002 ceph-mon[25764]: Adjusting osd_memory_target on smithi122 to 16064M 2024-03-11T21:48:50.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:49 smithi002 ceph-mon[25764]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:50.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:49 smithi002 ceph-mon[25764]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-11T21:48:50.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:49 smithi002 ceph-mon[25764]: Cluster is now healthy 2024-03-11T21:48:50.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:49 smithi002 ceph-mon[25764]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:48:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:49 smithi122 ceph-mon[27585]: Adjusting osd_memory_target on smithi122 to 16064M 2024-03-11T21:48:50.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:49 smithi122 ceph-mon[27585]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:48:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:49 smithi122 ceph-mon[27585]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-11T21:48:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:49 smithi122 ceph-mon[27585]: Cluster is now healthy 2024-03-11T21:48:50.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:49 smithi122 ceph-mon[27585]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:48:50.301 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:48:52.066 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:51 smithi165 ceph-mon[27942]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:48:52.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:51 smithi002 ceph-mon[25764]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:48:52.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:51 smithi122 ceph-mon[27585]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:48:53.037 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:52 smithi165 ceph-mon[27942]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:48:53.037 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:52 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:48:53.037 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:52 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:48:53.037 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:52 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:53.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:52 smithi122 ceph-mon[27585]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:48:53.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:52 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:48:53.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:52 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:48:53.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:52 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:53.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:52 smithi002 ceph-mon[25764]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:48:53.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:48:53.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:48:53.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:53.994 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:53 smithi165 ceph-mon[27942]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:48:54.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:53 smithi122 ceph-mon[27585]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:48:54.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:53 smithi002 ceph-mon[25764]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:48:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:54 smithi122 ceph-mon[27585]: from='client.? 172.21.15.165:0/2004999807' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8eb7bf8c-a48c-4f1e-9497-669b44d37c97"}]: dispatch 2024-03-11T21:48:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:54 smithi122 ceph-mon[27585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8eb7bf8c-a48c-4f1e-9497-669b44d37c97"}]: dispatch 2024-03-11T21:48:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:54 smithi122 ceph-mon[27585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8eb7bf8c-a48c-4f1e-9497-669b44d37c97"}]': finished 2024-03-11T21:48:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:54 smithi122 ceph-mon[27585]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:48:55.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:54 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:48:55.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:54 smithi002 ceph-mon[25764]: from='client.? 172.21.15.165:0/2004999807' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8eb7bf8c-a48c-4f1e-9497-669b44d37c97"}]: dispatch 2024-03-11T21:48:55.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:54 smithi002 ceph-mon[25764]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8eb7bf8c-a48c-4f1e-9497-669b44d37c97"}]: dispatch 2024-03-11T21:48:55.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:54 smithi002 ceph-mon[25764]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8eb7bf8c-a48c-4f1e-9497-669b44d37c97"}]': finished 2024-03-11T21:48:55.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:54 smithi002 ceph-mon[25764]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:48:55.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:54 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:48:55.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:54 smithi165 ceph-mon[27942]: from='client.? 172.21.15.165:0/2004999807' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8eb7bf8c-a48c-4f1e-9497-669b44d37c97"}]: dispatch 2024-03-11T21:48:55.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:54 smithi165 ceph-mon[27942]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8eb7bf8c-a48c-4f1e-9497-669b44d37c97"}]: dispatch 2024-03-11T21:48:55.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:54 smithi165 ceph-mon[27942]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8eb7bf8c-a48c-4f1e-9497-669b44d37c97"}]': finished 2024-03-11T21:48:55.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:54 smithi165 ceph-mon[27942]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:48:55.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:54 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:48:56.023 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:55 smithi002 ceph-mon[25764]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:48:56.023 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:55 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:56.023 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:55 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:56.023 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:55 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:56.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:55 smithi122 ceph-mon[27585]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:48:56.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:55 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:56.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:55 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:56.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:55 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:56.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:55 smithi165 ceph-mon[27942]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:48:56.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:55 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:56.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:55 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:56.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:55 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:48:57.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:56 smithi002 ceph-mon[25764]: from='client.? 172.21.15.165:0/3268027673' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:48:57.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:56 smithi122 ceph-mon[27585]: from='client.? 172.21.15.165:0/3268027673' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:48:57.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:56 smithi165 ceph-mon[27942]: from='client.? 172.21.15.165:0/3268027673' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:48:58.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:57 smithi165 ceph-mon[27942]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:48:58.173 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:57 smithi122 ceph-mon[27585]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:48:58.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:57 smithi002 ceph-mon[25764]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:00.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:59 smithi165 ceph-mon[27942]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:00.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:59 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:00.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:59 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:00.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:48:59 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:59 smithi122 ceph-mon[27585]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:59 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:59 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:00.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:48:59 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:00.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:59 smithi002 ceph-mon[25764]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:00.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:59 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:00.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:59 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:00.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:48:59 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:02.071 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:01 smithi122 ceph-mon[27585]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:02.071 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:01 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:02.071 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:01 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:02.071 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:01 smithi122 ceph-mon[27585]: Detected new or changed devices on smithi122 2024-03-11T21:49:02.071 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:01 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:02.296 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:01 smithi165 ceph-mon[27942]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:02.296 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:01 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:02.296 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:01 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:02.296 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:01 smithi165 ceph-mon[27942]: Detected new or changed devices on smithi122 2024-03-11T21:49:02.296 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:01 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:02.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:01 smithi002 ceph-mon[25764]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:02.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:01 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:02.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:01 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:02.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:01 smithi002 ceph-mon[25764]: Detected new or changed devices on smithi122 2024-03-11T21:49:02.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:01 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:04.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:03 smithi122 ceph-mon[27585]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:04.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:03 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:49:04.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:03 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:04.253 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:03 smithi165 ceph-mon[27942]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:04.253 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:03 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:49:04.253 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:03 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:04.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:03 smithi002 ceph-mon[25764]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:04.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:03 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:49:04.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:03 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:05.233 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:04 smithi165 ceph-mon[27942]: Deploying daemon osd.2 on smithi165 2024-03-11T21:49:05.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:04 smithi002 ceph-mon[25764]: Deploying daemon osd.2 on smithi165 2024-03-11T21:49:05.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:04 smithi122 ceph-mon[27585]: Deploying daemon osd.2 on smithi165 2024-03-11T21:49:06.214 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:05 smithi165 ceph-mon[27942]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:06.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:05 smithi002 ceph-mon[25764]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:06.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:05 smithi122 ceph-mon[27585]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:08.230 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:07 smithi165 ceph-mon[27942]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:08.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:07 smithi002 ceph-mon[25764]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:08.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:07 smithi122 ceph-mon[27585]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:10.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:10 smithi165 ceph-mon[27942]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:10.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:10 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:10.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:10 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:10.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:10 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:10.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:10 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:10.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:10 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:10.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:10 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:10.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:10 smithi002 ceph-mon[25764]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:10.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:10.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:10.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:10.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:10.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:10.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:10 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:10.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:10 smithi122 ceph-mon[27585]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:10.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:10 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:10.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:10 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:10.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:10 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:10.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:10 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:10.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:10 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:10.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:10 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:12.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:12 smithi165 ceph-mon[27942]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:12.150 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 2 on host 'smithi165' 2024-03-11T21:49:12.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:12 smithi002 ceph-mon[25764]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:12.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:12 smithi122 ceph-mon[27585]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:12.812 DEBUG:teuthology.orchestra.run.smithi165:osd.2> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T21:49:12.814 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-11T21:49:12.814 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T21:49:13.070 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:13.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:13 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:13.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:13 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:13.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:13 smithi002 ceph-mon[25764]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:13.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:13 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:13.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:13 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:49:13.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:13 smithi002 ceph-mon[25764]: Adjusting osd_memory_target on smithi165 to 16064M 2024-03-11T21:49:13.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:13 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:13.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:13 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:13.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:13 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:13.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:13 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:13.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:13 smithi002 ceph-mon[25764]: from='osd.2 [v2:172.21.15.165:6800/3979480837,v1:172.21.15.165:6801/3979480837]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:49:13.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:13 smithi002 ceph-mon[25764]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:49:13.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:13 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:13.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:13 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:13.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:13 smithi122 ceph-mon[27585]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:13.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:13 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:13.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:13 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:49:13.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:13 smithi122 ceph-mon[27585]: Adjusting osd_memory_target on smithi165 to 16064M 2024-03-11T21:49:13.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:13 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:13.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:13 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:13.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:13 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:13.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:13 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:13.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:13 smithi122 ceph-mon[27585]: from='osd.2 [v2:172.21.15.165:6800/3979480837,v1:172.21.15.165:6801/3979480837]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:49:13.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:13 smithi122 ceph-mon[27585]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:49:13.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:13 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:13.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:13 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:13.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:13 smithi165 ceph-mon[27942]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:13.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:13 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:13.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:13 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:49:13.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:13 smithi165 ceph-mon[27942]: Adjusting osd_memory_target on smithi165 to 16064M 2024-03-11T21:49:13.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:13 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:13.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:13 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:13.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:13 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:13.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:13 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:13.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:13 smithi165 ceph-mon[27942]: from='osd.2 [v2:172.21.15.165:6800/3979480837,v1:172.21.15.165:6801/3979480837]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:49:13.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:13 smithi165 ceph-mon[27942]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:49:14.262 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:14 smithi002 ceph-mon[25764]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:49:14.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:14 smithi002 ceph-mon[25764]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:49:14.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:14 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:14.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:14 smithi002 ceph-mon[25764]: from='osd.2 [v2:172.21.15.165:6800/3979480837,v1:172.21.15.165:6801/3979480837]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T21:49:14.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:14 smithi002 ceph-mon[25764]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T21:49:14.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:14 smithi122 ceph-mon[27585]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:49:14.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:14 smithi122 ceph-mon[27585]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:49:14.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:14 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:14.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:14 smithi122 ceph-mon[27585]: from='osd.2 [v2:172.21.15.165:6800/3979480837,v1:172.21.15.165:6801/3979480837]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T21:49:14.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:14 smithi122 ceph-mon[27585]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T21:49:14.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:14 smithi165 ceph-mon[27942]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:49:14.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:14 smithi165 ceph-mon[27942]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:49:14.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:14 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:14.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:14 smithi165 ceph-mon[27942]: from='osd.2 [v2:172.21.15.165:6800/3979480837,v1:172.21.15.165:6801/3979480837]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T21:49:14.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:14 smithi165 ceph-mon[27942]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T21:49:14.584 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:49:14 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-2[34500]: 2024-03-11T21:49:14.174+0000 7fb29f4af700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T21:49:14.584 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:49:14 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-2[34500]: 2024-03-11T21:49:14.180+0000 7fb29b2a2700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:49:15.181 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:49:15.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:15 smithi122 ceph-mon[27585]: purged_snaps scrub starts 2024-03-11T21:49:15.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:15 smithi122 ceph-mon[27585]: purged_snaps scrub ok 2024-03-11T21:49:15.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:15 smithi122 ceph-mon[27585]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:15.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:15 smithi122 ceph-mon[27585]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-11T21:49:15.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:15 smithi122 ceph-mon[27585]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:49:15.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:15 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:15.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:15 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:15.582 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:15 smithi165 ceph-mon[27942]: purged_snaps scrub starts 2024-03-11T21:49:15.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:15 smithi165 ceph-mon[27942]: purged_snaps scrub ok 2024-03-11T21:49:15.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:15 smithi165 ceph-mon[27942]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:15.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:15 smithi165 ceph-mon[27942]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-11T21:49:15.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:15 smithi165 ceph-mon[27942]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:49:15.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:15 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:15.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:15 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:15.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:15 smithi002 ceph-mon[25764]: purged_snaps scrub starts 2024-03-11T21:49:15.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:15 smithi002 ceph-mon[25764]: purged_snaps scrub ok 2024-03-11T21:49:15.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:15 smithi002 ceph-mon[25764]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:15.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:15 smithi002 ceph-mon[25764]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-11T21:49:15.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:15 smithi002 ceph-mon[25764]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:49:15.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:15 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:15.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:15 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:16.550 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":18,"num_osds":3,"num_up_osds":3,"osd_up_since":1710193755,"num_in_osds":3,"osd_in_since":1710193734,"num_remapped_pgs":0} 2024-03-11T21:49:16.550 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:49:16.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:16.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:16 smithi002 ceph-mon[25764]: osd.2 [v2:172.21.15.165:6800/3979480837,v1:172.21.15.165:6801/3979480837] boot 2024-03-11T21:49:16.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:16 smithi002 ceph-mon[25764]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T21:49:16.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:16.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:16 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1970838340' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:49:16.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:16.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:16.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:16.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:16 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:49:16.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:16 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:16.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:16 smithi122 ceph-mon[27585]: osd.2 [v2:172.21.15.165:6800/3979480837,v1:172.21.15.165:6801/3979480837] boot 2024-03-11T21:49:16.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:16 smithi122 ceph-mon[27585]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T21:49:16.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:16 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:16.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:16 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/1970838340' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:49:16.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:16 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:16.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:16 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:16.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:16 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:16.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:16 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:49:16.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:16 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:16.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:16 smithi165 ceph-mon[27942]: osd.2 [v2:172.21.15.165:6800/3979480837,v1:172.21.15.165:6801/3979480837] boot 2024-03-11T21:49:16.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:16 smithi165 ceph-mon[27942]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T21:49:16.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:16 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:16.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:16 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/1970838340' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:49:16.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:16 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:16.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:16 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:16.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:16 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:16.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:16 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:49:17.049 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:18.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:17 smithi122 ceph-mon[27585]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:18.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:17 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:49:18.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:17 smithi122 ceph-mon[27585]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:49:18.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:17 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:49:18.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:17 smithi002 ceph-mon[25764]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:18.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:17 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:49:18.236 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:17 smithi002 ceph-mon[25764]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:49:18.236 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:17 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:49:18.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:17 smithi165 ceph-mon[27942]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:49:18.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:17 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:49:18.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:17 smithi165 ceph-mon[27942]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:49:18.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:17 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:49:18.512 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:49:18 smithi002 sudo[40505]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T21:49:18.513 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:49:18 smithi002 sudo[40505]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:49:18.513 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:49:18 smithi002 sudo[40505]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:49:18.513 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:49:18 smithi002 sudo[40505]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:49:18.771 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:49:18 smithi002 sudo[40535]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T21:49:18.771 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:49:18 smithi002 sudo[40535]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:49:18.771 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:49:18 smithi002 sudo[40535]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:49:18.771 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:49:18 smithi002 sudo[40535]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:49:19.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:18 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:49:19.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:18 smithi122 ceph-mon[27585]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:49:19.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:18 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:19.035 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:49:18 smithi122 sudo[34723]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T21:49:19.035 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:49:18 smithi122 sudo[34723]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:49:19.035 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:49:18 smithi122 sudo[34723]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:49:19.035 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:49:18 smithi122 sudo[34723]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:49:19.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:18 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:49:19.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:18 smithi002 ceph-mon[25764]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:49:19.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:18 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:19.298 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:18 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:49:19.299 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:18 smithi165 ceph-mon[27942]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:49:19.299 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:18 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:19.424 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:49:19 smithi122 sudo[34726]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T21:49:19.424 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:49:19 smithi122 sudo[34726]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:49:19.424 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:49:19 smithi122 sudo[34726]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:49:19.424 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:49:19 smithi122 sudo[34726]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:49:19.427 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:49:19.428 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":21,"fsid":"85326acc-dff0-11ee-95c8-87774f69a715","created":"2024-03-11T21:45:41.992127+0000","modified":"2024-03-11T21:49:18.908564+0000","last_up_change":"2024-03-11T21:49:15.175277+0000","last_in_change":"2024-03-11T21:48:54.867543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:49:16.391641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac24d686-0fd0-411c-b9ea-6e2ea65d079d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":360299831},{"type":"v1","addr":"172.21.15.2:6803","nonce":360299831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":360299831},{"type":"v1","addr":"172.21.15.2:6805","nonce":360299831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":360299831},{"type":"v1","addr":"172.21.15.2:6809","nonce":360299831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":360299831},{"type":"v1","addr":"172.21.15.2:6807","nonce":360299831}]},"public_addr":"172.21.15.2:6803/360299831","cluster_addr":"172.21.15.2:6805/360299831","heartbeat_back_addr":"172.21.15.2:6809/360299831","heartbeat_front_addr":"172.21.15.2:6807/360299831","state":["exists","up"]},{"osd":1,"uuid":"126d0d98-285a-414f-a848-fdc47e6e3516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":377314645},{"type":"v1","addr":"172.21.15.122:6801","nonce":377314645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":377314645},{"type":"v1","addr":"172.21.15.122:6803","nonce":377314645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":377314645},{"type":"v1","addr":"172.21.15.122:6807","nonce":377314645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":377314645},{"type":"v1","addr":"172.21.15.122:6805","nonce":377314645}]},"public_addr":"172.21.15.122:6801/377314645","cluster_addr":"172.21.15.122:6803/377314645","heartbeat_back_addr":"172.21.15.122:6807/377314645","heartbeat_front_addr":"172.21.15.122:6805/377314645","state":["exists","up"]},{"osd":2,"uuid":"8eb7bf8c-a48c-4f1e-9497-669b44d37c97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3979480837},{"type":"v1","addr":"172.21.15.165:6801","nonce":3979480837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3979480837},{"type":"v1","addr":"172.21.15.165:6803","nonce":3979480837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3979480837},{"type":"v1","addr":"172.21.15.165:6807","nonce":3979480837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3979480837},{"type":"v1","addr":"172.21.15.165:6805","nonce":3979480837}]},"public_addr":"172.21.15.165:6801/3979480837","cluster_addr":"172.21.15.165:6803/3979480837","heartbeat_back_addr":"172.21.15.165:6807/3979480837","heartbeat_front_addr":"172.21.15.165:6805/3979480837","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:19.962641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:46.337955+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:49:13.187970+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/4068842992":"2024-03-12T21:46:42.041075+0000","172.21.15.2:0/2775318375":"2024-03-12T21:46:42.041075+0000","172.21.15.2:6801/751674141":"2024-03-12T21:46:15.215923+0000","172.21.15.2:6801/2572424991":"2024-03-12T21:46:42.041075+0000","172.21.15.2:6800/751674141":"2024-03-12T21:46:15.215923+0000","172.21.15.2:6800/2572424991":"2024-03-12T21:46:42.041075+0000","172.21.15.2:0/3448107164":"2024-03-12T21:46:42.041075+0000","172.21.15.2:0/3232348720":"2024-03-12T21:46:15.215923+0000","172.21.15.2:0/3938514137":"2024-03-12T21:46:15.215923+0000","172.21.15.2:0/592391913":"2024-03-12T21:46:15.215923+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:49:19.551 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:49:19 smithi165 sudo[35615]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T21:49:19.552 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:49:19 smithi165 sudo[35615]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:49:19.552 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:49:19 smithi165 sudo[35615]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:49:19.552 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:49:19 smithi165 sudo[35615]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:49:19.832 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:49:19 smithi165 sudo[35618]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T21:49:19.832 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:49:19 smithi165 sudo[35618]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:49:19.833 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:49:19 smithi165 sudo[35618]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:49:19.833 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:49:19 smithi165 sudo[35618]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:49:20.052 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:19 smithi002 sudo[40608]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:49:20.052 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:19 smithi002 sudo[40608]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:49:20.053 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:19 smithi002 sudo[40608]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:49:20.053 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:19 smithi002 sudo[40608]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:49:20.077 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T21:49:16.391641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T21:49:20.077 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T21:49:20.324 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:20.375 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 ceph-mon[27942]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:49:20.375 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 ceph-mon[27942]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:49:20.375 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:20.375 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:20.375 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 ceph-mon[27942]: Detected new or changed devices on smithi165 2024-03-11T21:49:20.375 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:20.376 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:20.376 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/2497515825' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:49:20.376 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:20.376 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 ceph-mon[27942]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:49:20.376 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 sudo[35635]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:49:20.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:20 smithi002 sudo[40627]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:49:20.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:20 smithi002 sudo[40627]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:49:20.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:20 smithi002 sudo[40627]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:49:20.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:20 smithi002 sudo[40627]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:49:20.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:20 smithi002 ceph-mon[25764]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:49:20.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:20 smithi002 ceph-mon[25764]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:49:20.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:20 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:20.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:20 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:20.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:20 smithi002 ceph-mon[25764]: Detected new or changed devices on smithi165 2024-03-11T21:49:20.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:20 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:20.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:20 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:20.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:20 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/2497515825' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:49:20.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:20 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:20.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:20 smithi002 ceph-mon[25764]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:49:20.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:20 smithi122 ceph-mon[27585]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:49:20.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:20 smithi122 ceph-mon[27585]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:49:20.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:20 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:20.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:20 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:20.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:20 smithi122 ceph-mon[27585]: Detected new or changed devices on smithi165 2024-03-11T21:49:20.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:20 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:20.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:20 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:20.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:20 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/2497515825' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:49:20.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:20 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:20.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:20 smithi122 ceph-mon[27585]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:49:20.634 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 sudo[35635]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:49:20.634 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 sudo[35635]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:49:20.634 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 sudo[35635]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:49:21.017 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 sudo[35658]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:49:21.017 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 sudo[35658]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:49:21.017 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 sudo[35658]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:49:21.017 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:20 smithi165 sudo[35658]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:49:21.145 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:20 smithi122 sudo[34797]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:49:21.145 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:20 smithi122 sudo[34797]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:49:21.145 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:20 smithi122 sudo[34797]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:49:21.145 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:20 smithi122 sudo[34797]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:49:21.268 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:49:21.269 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:49:21.269 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:49:21.269 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:49:21.269 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T21:49:21.269 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:21.269 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:21.269 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:21.269 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:21.269 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:49:21.269 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:49:21.269 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:49:21.269 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:49:21.269 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:49:21.270 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:49:21.270 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:21.270 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:49:21.270 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:49:21.270 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:21 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:49:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:49:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:49:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:49:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:49:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T21:49:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:21.367 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:21.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:21.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:21.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:49:21.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:49:21.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:49:21.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:49:21.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:49:21.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:49:21.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:21.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:49:21.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:49:21.368 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:21 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:49:21.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 sudo[34800]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:49:21.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 sudo[34800]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:49:21.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 sudo[34800]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:49:21.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 sudo[34800]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:49:21.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:49:21.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:49:21.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:49:21.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:49:21.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T21:49:21.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:21.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:21.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:21.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:21.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:49:21.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:49:21.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:49:21.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:49:21.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:49:21.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:49:21.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:21.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:49:21.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:49:21.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:21 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:49:22.129 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-03-11T21:49:22.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:22 smithi002 ceph-mon[25764]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:49:22.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:22 smithi002 ceph-mon[25764]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:49:22.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:22 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:22.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:22 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/177963948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:49:22.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:22 smithi122 ceph-mon[27585]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:49:22.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:22 smithi122 ceph-mon[27585]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:49:22.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:22 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:22.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:22 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/177963948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:49:22.495 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:22 smithi165 ceph-mon[27942]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:49:22.496 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:22 smithi165 ceph-mon[27942]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:49:22.496 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:22 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:22.496 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:22 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/177963948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:49:22.761 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi002 2024-03-11T21:49:22.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi002=foo.a' 2024-03-11T21:49:23.005 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:49:23.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:23 smithi122 ceph-mon[27585]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:49:23.428 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:23 smithi165 ceph-mon[27942]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:49:23.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:23 smithi002 ceph-mon[25764]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:49:24.773 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled rgw.foo.a update... 2024-03-11T21:49:25.351 DEBUG:teuthology.orchestra.run.smithi002:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-11T21:49:25.353 INFO:tasks.cephadm:Adding prometheus.a on smithi122 2024-03-11T21:49:25.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi122=a' 2024-03-11T21:49:25.587 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:49:26.000 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:25 smithi165 ceph-mon[27942]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 548 KiB/s wr, 35 op/s 2024-03-11T21:49:26.001 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:25 smithi165 ceph-mon[27942]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi002=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:26.001 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:25 smithi165 ceph-mon[27942]: Saving service rgw.foo.a spec with placement smithi002=foo.a;count:1 2024-03-11T21:49:26.001 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:25 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:26.001 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:25 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:26.001 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:25 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:26.001 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:25 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:26.001 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:25 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:26.001 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:25 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:49:26.001 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:25 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:49:26.002 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:25 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:26.002 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:25 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:26.002 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:25 smithi165 ceph-mon[27942]: Deploying daemon rgw.foo.a on smithi002 2024-03-11T21:49:26.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:25 smithi002 ceph-mon[25764]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 548 KiB/s wr, 35 op/s 2024-03-11T21:49:26.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:25 smithi002 ceph-mon[25764]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi002=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:26.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:25 smithi002 ceph-mon[25764]: Saving service rgw.foo.a spec with placement smithi002=foo.a;count:1 2024-03-11T21:49:26.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:26.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:26.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:26.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:26.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:26.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:49:26.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:49:26.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:26.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:25 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:26.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:25 smithi002 ceph-mon[25764]: Deploying daemon rgw.foo.a on smithi002 2024-03-11T21:49:26.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:25 smithi122 ceph-mon[27585]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 548 KiB/s wr, 35 op/s 2024-03-11T21:49:26.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:25 smithi122 ceph-mon[27585]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi002=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:26.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:25 smithi122 ceph-mon[27585]: Saving service rgw.foo.a spec with placement smithi002=foo.a;count:1 2024-03-11T21:49:26.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:25 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:26.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:25 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:26.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:25 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:26.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:25 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:26.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:25 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:26.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:25 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:49:26.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:25 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:49:26.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:25 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:26.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:25 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:26.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:25 smithi122 ceph-mon[27585]: Deploying daemon rgw.foo.a on smithi002 2024-03-11T21:49:27.145 INFO:journalctl@ceph.rgw.foo.a.smithi002.stdout:Mar 11 21:49:26 smithi002 systemd[1]: Starting Ceph rgw.foo.a for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:49:27.217 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled prometheus update... 2024-03-11T21:49:27.475 INFO:journalctl@ceph.rgw.foo.a.smithi002.stdout:Mar 11 21:49:27 smithi002 podman[41131]: 2024-03-11 21:49:27.087421138 +0000 UTC m=+0.017578137 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:49:27.475 INFO:journalctl@ceph.rgw.foo.a.smithi002.stdout:Mar 11 21:49:27 smithi002 podman[41131]: 2024-03-11 21:49:27.189992696 +0000 UTC m=+0.120149695 container create 9837194d9807bd57d8bb0a9bcb619d47f294204e7d1b5bea2224bccd7a0f944f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-rgw-foo-a, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-11T21:49:27.475 INFO:journalctl@ceph.rgw.foo.a.smithi002.stdout:Mar 11 21:49:27 smithi002 podman[41131]: 2024-03-11 21:49:27.409174442 +0000 UTC m=+0.339331454 container init 9837194d9807bd57d8bb0a9bcb619d47f294204e7d1b5bea2224bccd7a0f944f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_CLEAN=True, ceph=True) 2024-03-11T21:49:27.475 INFO:journalctl@ceph.rgw.foo.a.smithi002.stdout:Mar 11 21:49:27 smithi002 podman[41131]: 2024-03-11 21:49:27.414583287 +0000 UTC m=+0.344740287 container start 9837194d9807bd57d8bb0a9bcb619d47f294204e7d1b5bea2224bccd7a0f944f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:49:27.774 INFO:journalctl@ceph.rgw.foo.a.smithi002.stdout:Mar 11 21:49:27 smithi002 bash[41131]: 9837194d9807bd57d8bb0a9bcb619d47f294204e7d1b5bea2224bccd7a0f944f 2024-03-11T21:49:27.774 INFO:journalctl@ceph.rgw.foo.a.smithi002.stdout:Mar 11 21:49:27 smithi002 systemd[1]: Started Ceph rgw.foo.a for 85326acc-dff0-11ee-95c8-87774f69a715. 2024-03-11T21:49:27.828 DEBUG:teuthology.orchestra.run.smithi122:prometheus.a> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-11T21:49:27.830 INFO:tasks.cephadm:Adding node-exporter.a on smithi002 2024-03-11T21:49:27.831 INFO:tasks.cephadm:Adding node-exporter.b on smithi122 2024-03-11T21:49:27.831 INFO:tasks.cephadm:Adding node-exporter.c on smithi165 2024-03-11T21:49:27.831 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '3;smithi002=a;smithi122=b;smithi165=c' 2024-03-11T21:49:27.857 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:27 smithi165 ceph-mon[27942]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 397 KiB/s wr, 25 op/s 2024-03-11T21:49:27.858 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:27 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:27.858 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:27 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:27.858 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:27 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:27.858 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:27 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:27.858 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:27 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:27.858 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:27 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:28.066 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:49:28.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:27 smithi002 ceph-mon[25764]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 397 KiB/s wr, 25 op/s 2024-03-11T21:49:28.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:27 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:28.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:27 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:28.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:27 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:28.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:27 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:28.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:27 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:28.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:27 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:28.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:27 smithi122 ceph-mon[27585]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 7.5 KiB/s rd, 397 KiB/s wr, 25 op/s 2024-03-11T21:49:28.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:27 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:28.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:27 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:28.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:27 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:28.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:27 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:28.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:27 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:28.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:27 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:28.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:28 smithi165 ceph-mon[27942]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi122=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:28.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:28 smithi165 ceph-mon[27942]: Saving service prometheus spec with placement smithi122=a;count:1 2024-03-11T21:49:28.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:28 smithi165 ceph-mon[27942]: Saving service rgw.foo.a spec with placement smithi002=foo.a;count:1 2024-03-11T21:49:28.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:28 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:28.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:28 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:28.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:28 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:28.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:28 smithi165 ceph-mon[27942]: Checking dashboard <-> RGW credentials 2024-03-11T21:49:28.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:28 smithi165 ceph-mon[27942]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:49:28.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:28 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/3228370287' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:49:28.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:28 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/1762009102' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:49:29.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:28 smithi002 ceph-mon[25764]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi122=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:29.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:28 smithi002 ceph-mon[25764]: Saving service prometheus spec with placement smithi122=a;count:1 2024-03-11T21:49:29.043 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:28 smithi002 ceph-mon[25764]: Saving service rgw.foo.a spec with placement smithi002=foo.a;count:1 2024-03-11T21:49:29.043 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:28 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:29.043 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:28 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:29.043 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:28 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:29.043 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:28 smithi002 ceph-mon[25764]: Checking dashboard <-> RGW credentials 2024-03-11T21:49:29.043 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:28 smithi002 ceph-mon[25764]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:49:29.043 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:28 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/3228370287' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:49:29.043 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:28 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1762009102' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:49:29.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:28 smithi122 ceph-mon[27585]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi122=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:29.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:28 smithi122 ceph-mon[27585]: Saving service prometheus spec with placement smithi122=a;count:1 2024-03-11T21:49:29.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:28 smithi122 ceph-mon[27585]: Saving service rgw.foo.a spec with placement smithi002=foo.a;count:1 2024-03-11T21:49:29.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:28 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:29.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:28 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:29.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:28 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:29.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:28 smithi122 ceph-mon[27585]: Checking dashboard <-> RGW credentials 2024-03-11T21:49:29.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:28 smithi122 ceph-mon[27585]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:49:29.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:28 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/3228370287' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:49:29.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:28 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/1762009102' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:49:29.898 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:29 smithi002 ceph-mon[25764]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 5.8 KiB/s rd, 311 KiB/s wr, 19 op/s 2024-03-11T21:49:29.898 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:29 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/3228370287' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:49:29.898 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:29 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1762009102' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:49:29.898 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:29 smithi002 ceph-mon[25764]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T21:49:30.024 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled node-exporter update... 2024-03-11T21:49:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:29 smithi122 ceph-mon[27585]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 5.8 KiB/s rd, 311 KiB/s wr, 19 op/s 2024-03-11T21:49:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:29 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/3228370287' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:49:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:29 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/1762009102' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:49:30.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:29 smithi122 ceph-mon[27585]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T21:49:30.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:29 smithi165 ceph-mon[27942]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 5.8 KiB/s rd, 311 KiB/s wr, 19 op/s 2024-03-11T21:49:30.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:29 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/3228370287' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:49:30.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:29 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/1762009102' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:49:30.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:29 smithi165 ceph-mon[27942]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T21:49:30.547 DEBUG:teuthology.orchestra.run.smithi002:node-exporter.a> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-11T21:49:30.550 DEBUG:teuthology.orchestra.run.smithi122:node-exporter.b> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-11T21:49:30.553 DEBUG:teuthology.orchestra.run.smithi165:node-exporter.c> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@node-exporter.c.service 2024-03-11T21:49:30.590 INFO:tasks.cephadm:Adding alertmanager.a on smithi002 2024-03-11T21:49:30.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi002=a' 2024-03-11T21:49:30.864 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:49:31.264 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:31 smithi165 ceph-mon[27942]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi002=a;smithi122=b;smithi165=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:31.264 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:31 smithi165 ceph-mon[27942]: Saving service node-exporter spec with placement smithi002=a;smithi122=b;smithi165=c;count:3 2024-03-11T21:49:31.264 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:31 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:31.265 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:31 smithi165 ceph-mon[27942]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T21:49:31.265 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:31 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:49:31.265 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:31 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:49:31.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:31 smithi002 ceph-mon[25764]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi002=a;smithi122=b;smithi165=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:31.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:31 smithi002 ceph-mon[25764]: Saving service node-exporter spec with placement smithi002=a;smithi122=b;smithi165=c;count:3 2024-03-11T21:49:31.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:31 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:31.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:31 smithi002 ceph-mon[25764]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T21:49:31.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:31 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:49:31.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:31 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:49:31.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:31 smithi122 ceph-mon[27585]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi002=a;smithi122=b;smithi165=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:31.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:31 smithi122 ceph-mon[27585]: Saving service node-exporter spec with placement smithi002=a;smithi122=b;smithi165=c;count:3 2024-03-11T21:49:31.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:31 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:31.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:31 smithi122 ceph-mon[27585]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T21:49:31.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:31 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:49:31.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:31 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:49:32.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:32 smithi002 ceph-mon[25764]: pgmap v99: 33 pgs: 10 creating+peering, 22 unknown, 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 359 KiB/s wr, 22 op/s 2024-03-11T21:49:32.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:32 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:49:32.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:32 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:49:32.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:32 smithi002 ceph-mon[25764]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T21:49:32.263 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:32 smithi165 ceph-mon[27942]: pgmap v99: 33 pgs: 10 creating+peering, 22 unknown, 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 359 KiB/s wr, 22 op/s 2024-03-11T21:49:32.263 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:32 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:49:32.263 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:32 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:49:32.263 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:32 smithi165 ceph-mon[27942]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T21:49:32.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:32 smithi122 ceph-mon[27585]: pgmap v99: 33 pgs: 10 creating+peering, 22 unknown, 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 359 KiB/s wr, 22 op/s 2024-03-11T21:49:32.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:32 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:49:32.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:32 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:49:32.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:32 smithi122 ceph-mon[27585]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T21:49:32.689 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled alertmanager update... 2024-03-11T21:49:33.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:33 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:33 smithi165 ceph-mon[27942]: pgmap v102: 65 pgs: 19 creating+peering, 31 unknown, 15 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-11T21:49:33.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:33 smithi165 ceph-mon[27942]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T21:49:33.276 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:33 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:49:33.276 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:33 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:49:33.276 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:33 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.276 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:33 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.276 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:33 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.276 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:33 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.276 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:33 smithi165 ceph-mon[27942]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi002=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:33.276 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:33 smithi165 ceph-mon[27942]: Saving service alertmanager spec with placement smithi002=a;count:1 2024-03-11T21:49:33.276 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:33 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.277 DEBUG:teuthology.orchestra.run.smithi002:alertmanager.a> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-11T21:49:33.280 INFO:tasks.cephadm:Adding grafana.a on smithi122 2024-03-11T21:49:33.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi122=a' 2024-03-11T21:49:33.388 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:33 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.389 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:33 smithi002 ceph-mon[25764]: pgmap v102: 65 pgs: 19 creating+peering, 31 unknown, 15 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-11T21:49:33.389 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:33 smithi002 ceph-mon[25764]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T21:49:33.389 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:33 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:49:33.389 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:33 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:49:33.389 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:33 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.389 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:33 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.389 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:33 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.389 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:33 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.389 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:33 smithi002 ceph-mon[25764]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi002=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:33.389 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:33 smithi002 ceph-mon[25764]: Saving service alertmanager spec with placement smithi002=a;count:1 2024-03-11T21:49:33.389 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:33 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:33 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:33 smithi122 ceph-mon[27585]: pgmap v102: 65 pgs: 19 creating+peering, 31 unknown, 15 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-11T21:49:33.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:33 smithi122 ceph-mon[27585]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T21:49:33.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:33 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:49:33.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:33 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:49:33.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:33 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:33 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:33 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:33 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:33 smithi122 ceph-mon[27585]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi002=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:33.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:33 smithi122 ceph-mon[27585]: Saving service alertmanager spec with placement smithi002=a;count:1 2024-03-11T21:49:33.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:33 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:33.515 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:49:34.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:34 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:49:34.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:34 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:49:34.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:34 smithi165 ceph-mon[27942]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T21:49:34.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:34 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:49:34.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:34 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:49:34.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:34 smithi002 ceph-mon[25764]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T21:49:34.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:34 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:49:34.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:34 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:49:34.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:34 smithi122 ceph-mon[27585]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T21:49:35.359 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled grafana update... 2024-03-11T21:49:35.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:35 smithi165 ceph-mon[27942]: pgmap v105: 97 pgs: 20 creating+peering, 12 unknown, 65 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-11T21:49:35.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:35 smithi165 ceph-mon[27942]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T21:49:35.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:35 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:49:35.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:35 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:49:35.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:35 smithi002 ceph-mon[25764]: pgmap v105: 97 pgs: 20 creating+peering, 12 unknown, 65 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-11T21:49:35.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:35 smithi002 ceph-mon[25764]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T21:49:35.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:35 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:49:35.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:35 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:49:35.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:35 smithi122 ceph-mon[27585]: pgmap v105: 97 pgs: 20 creating+peering, 12 unknown, 65 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-11T21:49:35.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:35 smithi122 ceph-mon[27585]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T21:49:35.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:35 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:49:35.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:35 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:49:35.945 DEBUG:teuthology.orchestra.run.smithi122:grafana.a> sudo journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@grafana.a.service 2024-03-11T21:49:35.947 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T21:49:35.948 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T21:49:36.201 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:36.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:36 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:49:36.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:36 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:49:36.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:36 smithi002 ceph-mon[25764]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T21:49:36.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:36 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:49:36.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:36 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:49:36.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:36 smithi002 ceph-mon[25764]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi122=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:36.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:36 smithi002 ceph-mon[25764]: Saving service grafana spec with placement smithi122=a;count:1 2024-03-11T21:49:36.506 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:36 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:36.506 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:36 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:49:36.506 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:36 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:49:36.506 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:36 smithi002 ceph-mon[25764]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T21:49:36.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:36 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:49:36.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:36 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:49:36.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:36 smithi165 ceph-mon[27942]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T21:49:36.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:36 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:49:36.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:36 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:49:36.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:36 smithi165 ceph-mon[27942]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi122=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:36.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:36 smithi165 ceph-mon[27942]: Saving service grafana spec with placement smithi122=a;count:1 2024-03-11T21:49:36.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:36 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:36.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:36 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:49:36.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:36 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:49:36.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:36 smithi165 ceph-mon[27942]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T21:49:36.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:36 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:49:36.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:36 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:49:36.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:36 smithi122 ceph-mon[27585]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T21:49:36.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:36 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:49:36.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:36 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:49:36.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:36 smithi122 ceph-mon[27585]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi122=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:49:36.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:36 smithi122 ceph-mon[27585]: Saving service grafana spec with placement smithi122=a;count:1 2024-03-11T21:49:36.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:36 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:36.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:36 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/3940633759' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:49:36.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:36 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/1689600235' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:49:36.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:36 smithi122 ceph-mon[27585]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T21:49:37.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:37 smithi002 ceph-mon[25764]: pgmap v108: 129 pgs: 20 creating+peering, 44 unknown, 65 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T21:49:37.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:37 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:37.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:37 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:37.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:37 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:37.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:37 smithi002 ceph-mon[25764]: Deploying daemon prometheus.a on smithi122 2024-03-11T21:49:37.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:37 smithi122 ceph-mon[27585]: pgmap v108: 129 pgs: 20 creating+peering, 44 unknown, 65 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T21:49:37.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:37 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:37.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:37 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:37.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:37 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:37.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:37 smithi122 ceph-mon[27585]: Deploying daemon prometheus.a on smithi122 2024-03-11T21:49:37.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:37 smithi165 ceph-mon[27942]: pgmap v108: 129 pgs: 20 creating+peering, 44 unknown, 65 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T21:49:37.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:37 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:37.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:37 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:37.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:37 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:37.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:37 smithi165 ceph-mon[27942]: Deploying daemon prometheus.a on smithi122 2024-03-11T21:49:38.179 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-03-11T21:49:38.179 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQByfO9ljwggCxAAkl0H6TojiEl7Uo7M/lzi0Q== 2024-03-11T21:49:38.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:38 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/690537623' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:49:38.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:38 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/690537623' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:49:38.776 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T21:49:38.776 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T21:49:38.776 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T21:49:38.820 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T21:49:38.832 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:38 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/690537623' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:49:38.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:38 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/690537623' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:49:38.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:38 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/690537623' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:49:38.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:38 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/690537623' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:49:39.306 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:49:39.612 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:39 smithi122 ceph-mon[27585]: pgmap v110: 129 pgs: 13 unknown, 116 active+clean; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 35 KiB/s rd, 3.1 KiB/s wr, 85 op/s 2024-03-11T21:49:39.612 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:39 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:39.612 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:39 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:39.612 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:39 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:39.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:39 smithi165 ceph-mon[27942]: pgmap v110: 129 pgs: 13 unknown, 116 active+clean; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 35 KiB/s rd, 3.1 KiB/s wr, 85 op/s 2024-03-11T21:49:39.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:39 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:39.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:39 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:39.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:39 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:39.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:39 smithi002 ceph-mon[25764]: pgmap v110: 129 pgs: 13 unknown, 116 active+clean; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 35 KiB/s rd, 3.1 KiB/s wr, 85 op/s 2024-03-11T21:49:39.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:39 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:39.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:39 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:39.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:39 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:41.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:41 smithi165 ceph-mon[27942]: pgmap v111: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 55 KiB/s rd, 3.8 KiB/s wr, 133 op/s 2024-03-11T21:49:41.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:41 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:41.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:41 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:41.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:41 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:41.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:41 smithi002 ceph-mon[25764]: pgmap v111: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 55 KiB/s rd, 3.8 KiB/s wr, 133 op/s 2024-03-11T21:49:41.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:41 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:41.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:41 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:41.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:41 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:41.666 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:41 smithi122 ceph-mon[27585]: pgmap v111: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 55 KiB/s rd, 3.8 KiB/s wr, 133 op/s 2024-03-11T21:49:41.666 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:41 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:41.666 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:41 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:41.666 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:41 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:42.905 INFO:teuthology.orchestra.run.smithi122.stdout:[client.1] 2024-03-11T21:49:42.905 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQB2fO9lUvs9NhAAW8C+xxWKcYGkN1imKTm4sg== 2024-03-11T21:49:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:43 smithi165 ceph-mon[27942]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 141 op/s 2024-03-11T21:49:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:43 smithi165 ceph-mon[27942]: from='client.? 172.21.15.122:0/2713368135' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:49:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:43 smithi165 ceph-mon[27942]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:49:43.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:43 smithi165 ceph-mon[27942]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:49:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:43 smithi002 ceph-mon[25764]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 141 op/s 2024-03-11T21:49:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:43 smithi002 ceph-mon[25764]: from='client.? 172.21.15.122:0/2713368135' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:49:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:43 smithi002 ceph-mon[25764]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:49:43.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:43 smithi002 ceph-mon[25764]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:49:43.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:43 smithi122 ceph-mon[27585]: pgmap v112: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 141 op/s 2024-03-11T21:49:43.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:43 smithi122 ceph-mon[27585]: from='client.? 172.21.15.122:0/2713368135' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:49:43.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:43 smithi122 ceph-mon[27585]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:49:43.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:43 smithi122 ceph-mon[27585]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:49:44.450 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-11T21:49:44.450 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-11T21:49:44.450 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-11T21:49:44.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T21:49:44.785 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T21:49:45.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:45 smithi122 ceph-mon[27585]: pgmap v113: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 52 KiB/s rd, 3.9 KiB/s wr, 125 op/s 2024-03-11T21:49:45.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:45 smithi165 ceph-mon[27942]: pgmap v113: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 52 KiB/s rd, 3.9 KiB/s wr, 125 op/s 2024-03-11T21:49:45.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:45 smithi002 ceph-mon[25764]: pgmap v113: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 52 KiB/s rd, 3.9 KiB/s wr, 125 op/s 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi165.stdout:[client.2] 2024-03-11T21:49:46.882 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQB6fO9laf7WNBAA1LmC3DS028JMOG3zcNf3tQ== 2024-03-11T21:49:47.455 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:47 smithi165 ceph-mon[27942]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 111 op/s 2024-03-11T21:49:47.455 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:47 smithi165 ceph-mon[27942]: from='client.? 172.21.15.165:0/1937127002' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:49:47.455 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:47 smithi165 ceph-mon[27942]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:49:47.456 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:47 smithi165 ceph-mon[27942]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:49:47.456 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:47 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:47.483 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T21:49:47.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-11T21:49:47.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-11T21:49:47.526 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T21:49:47.526 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T21:49:47.526 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T21:49:47.554 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:47 smithi002 ceph-mon[25764]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 111 op/s 2024-03-11T21:49:47.554 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:47 smithi002 ceph-mon[25764]: from='client.? 172.21.15.165:0/1937127002' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:49:47.555 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:47 smithi002 ceph-mon[25764]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:49:47.555 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:47 smithi002 ceph-mon[25764]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:49:47.555 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:47 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:47.637 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:47 smithi122 ceph-mon[27585]: pgmap v114: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 111 op/s 2024-03-11T21:49:47.637 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:47 smithi122 ceph-mon[27585]: from='client.? 172.21.15.165:0/1937127002' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:49:47.637 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:47 smithi122 ceph-mon[27585]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:49:47.637 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:47 smithi122 ceph-mon[27585]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:49:47.637 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:47 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:47.776 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:48.379 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:48 smithi122 systemd[1]: Starting Ceph prometheus.a for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:49:48.379 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:48 smithi122 podman[35935]: 2024-03-11 21:49:48.290919055 +0000 UTC m=+0.018776312 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-11T21:49:49.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:49 smithi002 ceph-mon[25764]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 39 KiB/s rd, 2.9 KiB/s wr, 94 op/s 2024-03-11T21:49:49.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:49 smithi165 ceph-mon[27942]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 39 KiB/s rd, 2.9 KiB/s wr, 94 op/s 2024-03-11T21:49:49.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:49 smithi122 ceph-mon[27585]: pgmap v115: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 39 KiB/s rd, 2.9 KiB/s wr, 94 op/s 2024-03-11T21:49:49.674 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:49 smithi122 podman[35935]: 2024-03-11 21:49:49.24903201 +0000 UTC m=+0.976889262 container create 0becd75e25cc959b368ea5ce756cab8701c16b6d00d2a11f30d641f79f58220c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:49:49.962 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:49:50.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:50 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/3017455654' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:49:50.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:50 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/3017455654' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:49:50.587 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/3017455654' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:49:50.587 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 podman[35935]: 2024-03-11 21:49:50.260063877 +0000 UTC m=+1.987921130 container init 0becd75e25cc959b368ea5ce756cab8701c16b6d00d2a11f30d641f79f58220c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:49:50.587 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 podman[35935]: 2024-03-11 21:49:50.265811189 +0000 UTC m=+1.993668443 container start 0becd75e25cc959b368ea5ce756cab8701c16b6d00d2a11f30d641f79f58220c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:49:50.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.292Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T21:49:50.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.292Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T21:49:50.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.292Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi122 (none))" 2024-03-11T21:49:50.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.292Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T21:49:50.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.292Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T21:49:50.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.293Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T21:49:50.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.294Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T21:49:50.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.295Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T21:49:50.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.295Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T21:49:50.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.297Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T21:49:50.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.297Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.802µs 2024-03-11T21:49:50.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.297Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T21:49:50.589 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.298Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-11T21:49:50.589 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.298Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.986µs wal_replay_duration=276.835µs wbl_replay_duration=97ns total_replay_duration=313.271µs 2024-03-11T21:49:50.589 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.298Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T21:49:50.589 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.298Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T21:49:50.589 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.298Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T21:49:50.589 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.317Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.700921ms db_storage=804ns remote_storage=1.518µs web_handler=392ns query_engine=557ns scrape=499.767µs scrape_sd=93.493µs notify=1.129µs notify_sd=2.024µs rules=17.901064ms tracing=6.671µs 2024-03-11T21:49:50.589 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.317Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T21:49:50.589 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:49:50.317Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T21:49:50.589 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 bash[35935]: 0becd75e25cc959b368ea5ce756cab8701c16b6d00d2a11f30d641f79f58220c 2024-03-11T21:49:50.589 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:49:50 smithi122 systemd[1]: Started Ceph prometheus.a for 85326acc-dff0-11ee-95c8-87774f69a715. 2024-03-11T21:49:50.725 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3601168460},{"type":"v1","addr":"172.21.15.2:6801","nonce":3601168460}]},"active_addr":"172.21.15.2:6801/3601168460","active_change":"2024-03-11T21:46:42.041201+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.2:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":1875836275}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":438887713}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":1463310432}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":1888172482}]}]} 2024-03-11T21:49:50.730 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T21:49:50.730 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T21:49:50.730 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:49:51.612 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:51.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:51 smithi002 ceph-mon[25764]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 25 KiB/s rd, 1.8 KiB/s wr, 59 op/s 2024-03-11T21:49:51.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:51 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:51.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:51 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:51.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:51 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:51.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:51 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:49:51.646 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:51 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: ignoring --setuser ceph since I am not root 2024-03-11T21:49:51.646 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:51 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: ignoring --setgroup ceph since I am not root 2024-03-11T21:49:51.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:51 smithi122 ceph-mon[27585]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 25 KiB/s rd, 1.8 KiB/s wr, 59 op/s 2024-03-11T21:49:51.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:51 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:51.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:51 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:51.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:51 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:51.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:51 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:49:51.675 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:51 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: ignoring --setuser ceph since I am not root 2024-03-11T21:49:51.675 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:51 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: ignoring --setgroup ceph since I am not root 2024-03-11T21:49:51.675 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:51 smithi122 ceph-mgr[29267]: -- 172.21.15.122:0/2089532114 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55594604a1a0 con 0x555946020c00 2024-03-11T21:49:51.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:51 smithi165 ceph-mon[27942]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 25 KiB/s rd, 1.8 KiB/s wr, 59 op/s 2024-03-11T21:49:51.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:51 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:51.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:51 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:51.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:51 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' 2024-03-11T21:49:51.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:51 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:49:51.834 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:51 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: ignoring --setuser ceph since I am not root 2024-03-11T21:49:51.834 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:51 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: ignoring --setgroup ceph since I am not root 2024-03-11T21:49:51.834 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:51 smithi165 ceph-mgr[30399]: -- 172.21.15.165:0/815185761 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558c98cd61a0 con 0x558c98cad000 2024-03-11T21:49:52.082 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:51 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:51.997+0000 7f94616cd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:49:52.082 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:52 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:52.090+0000 7f94616cd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:49:52.280 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:52 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:52.055+0000 7fefca82e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:49:52.280 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:52 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:52.157+0000 7fefca82e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:49:52.333 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:52 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:52.003+0000 7f88c3ac7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:49:52.333 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:52 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:52.097+0000 7f88c3ac7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:49:52.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:52 smithi002 ceph-mon[25764]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:49:52.559 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:52 smithi002 ceph-mon[25764]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T21:49:52.716 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:52 smithi122 ceph-mon[27585]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:49:52.716 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:52 smithi122 ceph-mon[27585]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T21:49:52.733 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:52 smithi165 ceph-mon[27942]: from='mgr.14152 172.21.15.2:0/985350496' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:49:52.733 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:52 smithi165 ceph-mon[27942]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T21:49:52.992 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:52 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:52.751+0000 7f88c3ac7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:49:52.992 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:52 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:52.850+0000 7f88c3ac7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:49:52.992 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:53 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:52.999+0000 7f88c3ac7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:49:53.044 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:52 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:52.724+0000 7f94616cd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:49:53.045 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:52 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:52.817+0000 7f94616cd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:49:53.045 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:52 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:52.958+0000 7f94616cd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:49:53.045 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:53 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:53.052+0000 7f94616cd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:49:53.145 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:52 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:52.826+0000 7fefca82e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:49:53.145 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:52 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:52.925+0000 7fefca82e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:49:53.145 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:53 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:53.069+0000 7fefca82e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:49:53.333 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:53 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:53.099+0000 7f88c3ac7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:49:53.333 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:53 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:53.186+0000 7f88c3ac7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:49:53.388 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:53 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:53.135+0000 7f94616cd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:49:53.388 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:53 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:53.396+0000 7f94616cd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:49:53.462 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:53 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:53.167+0000 7fefca82e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:49:53.462 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:53 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:53.254+0000 7fefca82e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:49:53.674 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:53 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:53.506+0000 7f94616cd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:49:53.674 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:53 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:53.593+0000 7f94616cd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:49:53.730 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:53 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:53.530+0000 7fefca82e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:49:53.731 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:53 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:53.648+0000 7fefca82e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:49:53.731 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:53 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:53.738+0000 7fefca82e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:49:53.751 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:53 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:53.459+0000 7f88c3ac7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:49:53.751 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:53 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:53.572+0000 7f88c3ac7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:49:53.751 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:53 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:53.660+0000 7f88c3ac7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:49:54.016 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:53 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:53.843+0000 7fefca82e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:49:54.016 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:54 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:54.023+0000 7fefca82e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:49:54.020 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:49:54.021 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":30,"fsid":"85326acc-dff0-11ee-95c8-87774f69a715","created":"2024-03-11T21:45:41.992127+0000","modified":"2024-03-11T21:49:36.252520+0000","last_up_change":"2024-03-11T21:49:15.175277+0000","last_in_change":"2024-03-11T21:48:54.867543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:49:16.391641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T21:49:27.703742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:49:29.569569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:49:31.500051+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:49:33.563340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac24d686-0fd0-411c-b9ea-6e2ea65d079d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":360299831},{"type":"v1","addr":"172.21.15.2:6803","nonce":360299831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":360299831},{"type":"v1","addr":"172.21.15.2:6805","nonce":360299831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":360299831},{"type":"v1","addr":"172.21.15.2:6809","nonce":360299831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":360299831},{"type":"v1","addr":"172.21.15.2:6807","nonce":360299831}]},"public_addr":"172.21.15.2:6803/360299831","cluster_addr":"172.21.15.2:6805/360299831","heartbeat_back_addr":"172.21.15.2:6809/360299831","heartbeat_front_addr":"172.21.15.2:6807/360299831","state":["exists","up"]},{"osd":1,"uuid":"126d0d98-285a-414f-a848-fdc47e6e3516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":377314645},{"type":"v1","addr":"172.21.15.122:6801","nonce":377314645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":377314645},{"type":"v1","addr":"172.21.15.122:6803","nonce":377314645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":377314645},{"type":"v1","addr":"172.21.15.122:6807","nonce":377314645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":377314645},{"type":"v1","addr":"172.21.15.122:6805","nonce":377314645}]},"public_addr":"172.21.15.122:6801/377314645","cluster_addr":"172.21.15.122:6803/377314645","heartbeat_back_addr":"172.21.15.122:6807/377314645","heartbeat_front_addr":"172.21.15.122:6805/377314645","state":["exists","up"]},{"osd":2,"uuid":"8eb7bf8c-a48c-4f1e-9497-669b44d37c97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3979480837},{"type":"v1","addr":"172.21.15.165:6801","nonce":3979480837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3979480837},{"type":"v1","addr":"172.21.15.165:6803","nonce":3979480837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3979480837},{"type":"v1","addr":"172.21.15.165:6807","nonce":3979480837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3979480837},{"type":"v1","addr":"172.21.15.165:6805","nonce":3979480837}]},"public_addr":"172.21.15.165:6801/3979480837","cluster_addr":"172.21.15.165:6803/3979480837","heartbeat_back_addr":"172.21.15.165:6807/3979480837","heartbeat_front_addr":"172.21.15.165:6805/3979480837","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:19.962641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:46.337955+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:49:13.187970+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/4068842992":"2024-03-12T21:46:42.041075+0000","172.21.15.2:0/2775318375":"2024-03-12T21:46:42.041075+0000","172.21.15.2:6801/751674141":"2024-03-12T21:46:15.215923+0000","172.21.15.2:6801/2572424991":"2024-03-12T21:46:42.041075+0000","172.21.15.2:6800/751674141":"2024-03-12T21:46:15.215923+0000","172.21.15.2:6800/2572424991":"2024-03-12T21:46:42.041075+0000","172.21.15.2:0/3448107164":"2024-03-12T21:46:42.041075+0000","172.21.15.2:0/3232348720":"2024-03-12T21:46:15.215923+0000","172.21.15.2:0/3938514137":"2024-03-12T21:46:15.215923+0000","172.21.15.2:0/592391913":"2024-03-12T21:46:15.215923+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:49:54.072 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:53 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:53.687+0000 7f94616cd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:49:54.072 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:53 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:53.862+0000 7f94616cd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:49:54.073 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:53 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:53.758+0000 7f88c3ac7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:49:54.073 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:53 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:53.937+0000 7f88c3ac7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:49:54.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:54 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/3598160488' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:49:54.384 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:54 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/3598160488' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:49:54.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:54 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/3598160488' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:49:54.669 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T21:49:54.670 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:49:54.915 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:55.174 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:54 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:54.817+0000 7f94616cd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:49:55.174 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:54 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:54.947+0000 7f94616cd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:49:55.303 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:55.043+0000 7fefca82e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:49:55.303 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:55.181+0000 7fefca82e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:49:55.332 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:54 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:54.904+0000 7f88c3ac7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:49:55.333 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:55 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:55.035+0000 7f88c3ac7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:49:55.557 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:55 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:55.278+0000 7f94616cd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:49:55.557 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:55 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:55.368+0000 7f94616cd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:49:55.557 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:55 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:55.565+0000 7f94616cd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:49:55.614 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:55.528+0000 7fefca82e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:49:55.653 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:55 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:55.373+0000 7f88c3ac7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:49:55.653 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:55 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:55.463+0000 7f88c3ac7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:49:55.653 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:55 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:55.661+0000 7f88c3ac7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:49:55.895 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:55.622+0000 7fefca82e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:49:55.895 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:55.827+0000 7fefca82e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:49:55.924 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:55 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:55.749+0000 7f94616cd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:49:56.083 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:55 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:55.845+0000 7f88c3ac7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:49:56.285 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:56 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:56.034+0000 7f94616cd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:49:56.322 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:56 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:56.019+0000 7fefca82e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:49:56.387 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:56 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:56.132+0000 7f88c3ac7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:49:56.388 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:56 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:56.395+0000 7f88c3ac7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:49:56.593 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:56 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:56.329+0000 7fefca82e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:49:56.593 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:56 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:56.600+0000 7fefca82e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:49:56.674 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:56 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:56.293+0000 7f94616cd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:49:56.674 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:56 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:56.542+0000 7f94616cd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:49:56.751 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:49:56.751 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":30,"fsid":"85326acc-dff0-11ee-95c8-87774f69a715","created":"2024-03-11T21:45:41.992127+0000","modified":"2024-03-11T21:49:36.252520+0000","last_up_change":"2024-03-11T21:49:15.175277+0000","last_in_change":"2024-03-11T21:48:54.867543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:49:16.391641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T21:49:27.703742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:49:29.569569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:49:31.500051+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:49:33.563340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ac24d686-0fd0-411c-b9ea-6e2ea65d079d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":360299831},{"type":"v1","addr":"172.21.15.2:6803","nonce":360299831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":360299831},{"type":"v1","addr":"172.21.15.2:6805","nonce":360299831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":360299831},{"type":"v1","addr":"172.21.15.2:6809","nonce":360299831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":360299831},{"type":"v1","addr":"172.21.15.2:6807","nonce":360299831}]},"public_addr":"172.21.15.2:6803/360299831","cluster_addr":"172.21.15.2:6805/360299831","heartbeat_back_addr":"172.21.15.2:6809/360299831","heartbeat_front_addr":"172.21.15.2:6807/360299831","state":["exists","up"]},{"osd":1,"uuid":"126d0d98-285a-414f-a848-fdc47e6e3516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":377314645},{"type":"v1","addr":"172.21.15.122:6801","nonce":377314645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":377314645},{"type":"v1","addr":"172.21.15.122:6803","nonce":377314645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":377314645},{"type":"v1","addr":"172.21.15.122:6807","nonce":377314645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":377314645},{"type":"v1","addr":"172.21.15.122:6805","nonce":377314645}]},"public_addr":"172.21.15.122:6801/377314645","cluster_addr":"172.21.15.122:6803/377314645","heartbeat_back_addr":"172.21.15.122:6807/377314645","heartbeat_front_addr":"172.21.15.122:6805/377314645","state":["exists","up"]},{"osd":2,"uuid":"8eb7bf8c-a48c-4f1e-9497-669b44d37c97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":3979480837},{"type":"v1","addr":"172.21.15.165:6801","nonce":3979480837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3979480837},{"type":"v1","addr":"172.21.15.165:6803","nonce":3979480837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3979480837},{"type":"v1","addr":"172.21.15.165:6807","nonce":3979480837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3979480837},{"type":"v1","addr":"172.21.15.165:6805","nonce":3979480837}]},"public_addr":"172.21.15.165:6801/3979480837","cluster_addr":"172.21.15.165:6803/3979480837","heartbeat_back_addr":"172.21.15.165:6807/3979480837","heartbeat_front_addr":"172.21.15.165:6805/3979480837","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:19.962641+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:48:46.337955+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:49:13.187970+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/4068842992":"2024-03-12T21:46:42.041075+0000","172.21.15.2:0/2775318375":"2024-03-12T21:46:42.041075+0000","172.21.15.2:6801/751674141":"2024-03-12T21:46:15.215923+0000","172.21.15.2:6801/2572424991":"2024-03-12T21:46:42.041075+0000","172.21.15.2:6800/751674141":"2024-03-12T21:46:15.215923+0000","172.21.15.2:6800/2572424991":"2024-03-12T21:46:42.041075+0000","172.21.15.2:0/3448107164":"2024-03-12T21:46:42.041075+0000","172.21.15.2:0/3232348720":"2024-03-12T21:46:15.215923+0000","172.21.15.2:0/3938514137":"2024-03-12T21:46:15.215923+0000","172.21.15.2:0/592391913":"2024-03-12T21:46:15.215923+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:49:56.802 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:56 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:56.632+0000 7f88c3ac7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:49:56.844 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:56 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1556986234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:49:57.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:56 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/1556986234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:49:57.145 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:56 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:56.852+0000 7fefca82e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:49:57.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:56 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/1556986234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:49:57.391 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T21:49:57.392 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T21:49:57.392 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T21:49:57.674 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:57 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:57.376+0000 7f94616cd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:49:57.681 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:57.685 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:57.686 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:57.833 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:57 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:57.436+0000 7f88c3ac7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:49:58.016 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:57 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:57.750+0000 7fefca82e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:49:58.254 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:57 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:57.948+0000 7f94616cd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:49:58.255 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:58 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:58.047+0000 7f94616cd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:49:58.317 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:58 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:58.012+0000 7f88c3ac7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:49:58.317 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:58 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:58.110+0000 7f88c3ac7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:49:58.317 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:58 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:58.325+0000 7f88c3ac7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:49:58.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:58 smithi165 ceph-mon[27942]: Standby manager daemon x restarted 2024-03-11T21:49:58.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:58 smithi165 ceph-mon[27942]: Standby manager daemon x started 2024-03-11T21:49:58.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:58 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.122:0/3576214466' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:49:58.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:58 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.122:0/3576214466' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:49:58.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:58 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.122:0/3576214466' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:49:58.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:58 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.122:0/3576214466' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:49:58.583 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:58 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: [11/Mar/2024:21:49:58] ENGINE Bus STARTING 2024-03-11T21:49:58.584 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:58 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: CherryPy Checker: 2024-03-11T21:49:58.584 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:58 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: The Application mounted at '' has an empty config. 2024-03-11T21:49:58.584 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:58 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: 2024-03-11T21:49:58.584 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:58 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: [11/Mar/2024:21:49:58] ENGINE Serving on http://:::9283 2024-03-11T21:49:58.584 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:49:58 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z[30340]: [11/Mar/2024:21:49:58] ENGINE Bus STARTED 2024-03-11T21:49:58.645 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:58 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:58.351+0000 7fefca82e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:49:58.645 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:58 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:58.455+0000 7fefca82e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:49:58.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:58 smithi002 ceph-mon[25764]: Standby manager daemon x restarted 2024-03-11T21:49:58.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:58 smithi002 ceph-mon[25764]: Standby manager daemon x started 2024-03-11T21:49:58.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:58 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.122:0/3576214466' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:49:58.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:58 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.122:0/3576214466' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:49:58.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:58 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.122:0/3576214466' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:49:58.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:58 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.122:0/3576214466' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:49:58.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:58 smithi122 ceph-mon[27585]: Standby manager daemon x restarted 2024-03-11T21:49:58.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:58 smithi122 ceph-mon[27585]: Standby manager daemon x started 2024-03-11T21:49:58.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:58 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.122:0/3576214466' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:49:58.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:58 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.122:0/3576214466' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:49:58.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:58 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.122:0/3576214466' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:49:58.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:58 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.122:0/3576214466' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:49:58.675 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:58 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:58.262+0000 7f94616cd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:49:58.675 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:58 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: [11/Mar/2024:21:49:58] ENGINE Bus STARTING 2024-03-11T21:49:58.675 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:58 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: CherryPy Checker: 2024-03-11T21:49:58.675 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:58 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: The Application mounted at '' has an empty config. 2024-03-11T21:49:58.675 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:58 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: 2024-03-11T21:49:58.675 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:58 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: [11/Mar/2024:21:49:58] ENGINE Serving on http://:::9283 2024-03-11T21:49:58.675 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:49:58 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x[29243]: [11/Mar/2024:21:49:58] ENGINE Bus STARTED 2024-03-11T21:49:59.012 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:58 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:58.681+0000 7fefca82e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:49:59.012 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:58 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:49:58] ENGINE Bus STARTING 2024-03-11T21:49:59.012 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:58 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: CherryPy Checker: 2024-03-11T21:49:59.012 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:58 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: The Application mounted at '' has an empty config. 2024-03-11T21:49:59.012 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:58 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: 2024-03-11T21:49:59.013 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:58 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:49:58] ENGINE Serving on http://:::9283 2024-03-11T21:49:59.013 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:49:58 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:49:58] ENGINE Bus STARTED 2024-03-11T21:49:59.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: mgrmap e18: y(active, since 3m), standbys: z, x 2024-03-11T21:49:59.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: Standby manager daemon z restarted 2024-03-11T21:49:59.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.165:0/1693850183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:49:59.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: Standby manager daemon z started 2024-03-11T21:49:59.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.165:0/1693850183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:49:59.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.165:0/1693850183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:49:59.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.? 172.21.15.165:0/1693850183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:49:59.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: Active manager daemon y restarted 2024-03-11T21:49:59.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: Activating manager daemon y 2024-03-11T21:49:59.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T21:49:59.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: mgrmap e19: y(active, starting, since 0.010208s), standbys: x, z 2024-03-11T21:49:59.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:49:59.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:49:59.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:49:59.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:49:59.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:49:59.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:49:59.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:49:59.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:49:59.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:59.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:49:59.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:49:59.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:49:59.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: Manager daemon y is now available 2024-03-11T21:49:59.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:49:59.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]': finished 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]': finished 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]': finished 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:49:59 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:49:59.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: mgrmap e18: y(active, since 3m), standbys: z, x 2024-03-11T21:49:59.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: Standby manager daemon z restarted 2024-03-11T21:49:59.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.165:0/1693850183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:49:59.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: Standby manager daemon z started 2024-03-11T21:49:59.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.165:0/1693850183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:49:59.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.165:0/1693850183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:49:59.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.? 172.21.15.165:0/1693850183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:49:59.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: Active manager daemon y restarted 2024-03-11T21:49:59.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: Activating manager daemon y 2024-03-11T21:49:59.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T21:49:59.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: mgrmap e19: y(active, starting, since 0.010208s), standbys: x, z 2024-03-11T21:49:59.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:49:59.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:49:59.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:49:59.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:49:59.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:49:59.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:49:59.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:49:59.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:49:59.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:59.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:49:59.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:49:59.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:49:59.648 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: Manager daemon y is now available 2024-03-11T21:49:59.648 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:49:59.648 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:59.648 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:59.648 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:49:59.648 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:49:59.648 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]': finished 2024-03-11T21:49:59.648 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:49:59.648 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]': finished 2024-03-11T21:49:59.648 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]': finished 2024-03-11T21:49:59.648 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:49:59.648 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:59.648 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:59.649 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:59.649 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:49:59.649 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:49:59 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:49:59.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: mgrmap e18: y(active, since 3m), standbys: z, x 2024-03-11T21:49:59.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: Standby manager daemon z restarted 2024-03-11T21:49:59.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.165:0/1693850183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T21:49:59.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: Standby manager daemon z started 2024-03-11T21:49:59.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.165:0/1693850183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:49:59.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.165:0/1693850183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T21:49:59.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.? 172.21.15.165:0/1693850183' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:49:59.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: Active manager daemon y restarted 2024-03-11T21:49:59.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: Activating manager daemon y 2024-03-11T21:49:59.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T21:49:59.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: mgrmap e19: y(active, starting, since 0.010208s), standbys: x, z 2024-03-11T21:49:59.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:49:59.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:49:59.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:49:59.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:49:59.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:49:59.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T21:49:59.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:49:59.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:49:59.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:49:59.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:49:59.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:49:59.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:49:59.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: Manager daemon y is now available 2024-03-11T21:49:59.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:49:59.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:59.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:59.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:49:59.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:49:59.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]': finished 2024-03-11T21:49:59.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:49:59.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]': finished 2024-03-11T21:49:59.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]': finished 2024-03-11T21:49:59.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:49:59.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:59.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:59.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:59.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:49:59.678 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:49:59 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:50:00.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: Updating smithi002:/etc/ceph/ceph.conf 2024-03-11T21:50:00.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-11T21:50:00.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-11T21:50:00.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: [11/Mar/2024:21:49:59] ENGINE Bus STARTING 2024-03-11T21:50:00.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: [11/Mar/2024:21:49:59] ENGINE Serving on http://172.21.15.2:8765 2024-03-11T21:50:00.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: [11/Mar/2024:21:49:59] ENGINE Serving on https://172.21.15.2:7151 2024-03-11T21:50:00.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: [11/Mar/2024:21:49:59] ENGINE Bus STARTED 2024-03-11T21:50:00.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: Updating smithi165:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:50:00.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: Updating smithi122:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:50:00.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: Updating smithi002:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:50:00.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: mgrmap e20: y(active, since 1.01628s), standbys: x, z 2024-03-11T21:50:00.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: overall HEALTH_OK 2024-03-11T21:50:00.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:00 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi002", "caps": []}]: dispatch 2024-03-11T21:50:00.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: Updating smithi002:/etc/ceph/ceph.conf 2024-03-11T21:50:00.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-11T21:50:00.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-11T21:50:00.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: [11/Mar/2024:21:49:59] ENGINE Bus STARTING 2024-03-11T21:50:00.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: [11/Mar/2024:21:49:59] ENGINE Serving on http://172.21.15.2:8765 2024-03-11T21:50:00.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: [11/Mar/2024:21:49:59] ENGINE Serving on https://172.21.15.2:7151 2024-03-11T21:50:00.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: [11/Mar/2024:21:49:59] ENGINE Bus STARTED 2024-03-11T21:50:00.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: Updating smithi165:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:50:00.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: Updating smithi122:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:50:00.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: Updating smithi002:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:50:00.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: mgrmap e20: y(active, since 1.01628s), standbys: x, z 2024-03-11T21:50:00.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: overall HEALTH_OK 2024-03-11T21:50:00.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:00 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi002", "caps": []}]: dispatch 2024-03-11T21:50:00.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: Updating smithi002:/etc/ceph/ceph.conf 2024-03-11T21:50:00.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-11T21:50:00.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-11T21:50:00.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: [11/Mar/2024:21:49:59] ENGINE Bus STARTING 2024-03-11T21:50:00.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: [11/Mar/2024:21:49:59] ENGINE Serving on http://172.21.15.2:8765 2024-03-11T21:50:00.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: [11/Mar/2024:21:49:59] ENGINE Serving on https://172.21.15.2:7151 2024-03-11T21:50:00.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: [11/Mar/2024:21:49:59] ENGINE Bus STARTED 2024-03-11T21:50:00.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: Updating smithi165:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:50:00.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: Updating smithi122:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:50:00.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: Updating smithi002:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:50:00.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: mgrmap e20: y(active, since 1.01628s), standbys: x, z 2024-03-11T21:50:00.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: overall HEALTH_OK 2024-03-11T21:50:00.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:00.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:00 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi002", "caps": []}]: dispatch 2024-03-11T21:50:01.550 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:01 smithi002 ceph-mon[25764]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:01.550 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:01 smithi002 ceph-mon[25764]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:01.550 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:01 smithi002 ceph-mon[25764]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:01.550 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:01 smithi002 ceph-mon[25764]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:01.551 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:01 smithi002 ceph-mon[25764]: Updating smithi165:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:50:01.551 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:01 smithi002 ceph-mon[25764]: Updating smithi122:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:50:01.551 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:01 smithi002 ceph-mon[25764]: Updating smithi002:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:50:01.551 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:01 smithi002 ceph-mon[25764]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:50:01.551 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:01 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi122", "caps": []}]: dispatch 2024-03-11T21:50:01.551 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:01 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]: dispatch 2024-03-11T21:50:01.582 INFO:teuthology.orchestra.run.smithi002.stdout:77309411340 2024-03-11T21:50:01.583 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T21:50:01.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:01 smithi165 ceph-mon[27942]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:01.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:01 smithi165 ceph-mon[27942]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:01.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:01 smithi165 ceph-mon[27942]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:01.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:01 smithi165 ceph-mon[27942]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:01.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:01 smithi165 ceph-mon[27942]: Updating smithi165:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:50:01.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:01 smithi165 ceph-mon[27942]: Updating smithi122:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:50:01.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:01 smithi165 ceph-mon[27942]: Updating smithi002:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:50:01.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:01 smithi165 ceph-mon[27942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:50:01.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:01 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi122", "caps": []}]: dispatch 2024-03-11T21:50:01.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:01 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]: dispatch 2024-03-11T21:50:01.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:01 smithi122 ceph-mon[27585]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:01.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:01 smithi122 ceph-mon[27585]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:01.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:01 smithi122 ceph-mon[27585]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:01.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:01 smithi122 ceph-mon[27585]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:01.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:01 smithi122 ceph-mon[27585]: Updating smithi165:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:50:01.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:01 smithi122 ceph-mon[27585]: Updating smithi122:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:50:01.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:01 smithi122 ceph-mon[27585]: Updating smithi002:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:50:01.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:01 smithi122 ceph-mon[27585]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:50:01.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:01 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi122", "caps": []}]: dispatch 2024-03-11T21:50:01.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:01 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]: dispatch 2024-03-11T21:50:02.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:02 smithi002 ceph-mon[25764]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:02.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:02 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:02.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:02 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:02.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:02 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:02.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:02 smithi002 ceph-mon[25764]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-11T21:50:02.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:02 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:02.550 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:02 smithi122 ceph-mon[27585]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:02.550 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:02 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:02.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:02 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:02.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:02 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:02.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:02 smithi122 ceph-mon[27585]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-11T21:50:02.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:02 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:02.554 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:02 smithi165 ceph-mon[27942]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:02.554 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:02 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:02.554 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:02 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:02.554 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:02 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:02.554 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:02 smithi165 ceph-mon[27942]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-11T21:50:02.554 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:02 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:03 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:03 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:03.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:03 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]: dispatch 2024-03-11T21:50:03.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:03 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:03.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:03 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:03.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:03 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:03 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:03 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:03 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:03 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:03 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.690 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:03 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.690 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:03 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:03.690 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:03 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]: dispatch 2024-03-11T21:50:03.690 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:03 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:03.690 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:03 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:03.690 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:03 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.690 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:03 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.691 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:03 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.691 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:03 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.691 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:03 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.691 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:03 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.742 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:03 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.742 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:03 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:03.742 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:03 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi165", "caps": []}]: dispatch 2024-03-11T21:50:03.742 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:03 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:03.742 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:03 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:03.742 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:03 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.742 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:03 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.742 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:03 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.742 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:03 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.743 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:03 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.743 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:03 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:03.993 INFO:teuthology.orchestra.run.smithi002.stdout:34359738389 2024-03-11T21:50:03.993 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T21:50:04.269 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:04 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: ::ffff:172.21.15.122 - - [11/Mar/2024:21:50:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:04.642 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:04 smithi165 ceph-mon[27942]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:04.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:04 smithi165 ceph-mon[27942]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:50:04.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:04 smithi165 ceph-mon[27942]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-11T21:50:04.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:04 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:04 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:04 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:04 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:04 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:04 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:04 smithi002 ceph-mon[25764]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:04.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:04 smithi002 ceph-mon[25764]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:50:04.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:04 smithi002 ceph-mon[25764]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-11T21:50:04.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:04 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:04 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:04 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:04 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:04 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:04 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:04 smithi122 ceph-mon[27585]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:04.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:04 smithi122 ceph-mon[27585]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T21:50:04.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:04 smithi122 ceph-mon[27585]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-11T21:50:04.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:04 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:04 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:04 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:04 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:04 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:04.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:04 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.031 INFO:teuthology.orchestra.run.smithi002.stdout:55834574865 2024-03-11T21:50:05.031 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T21:50:05.421 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:05.469 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:05.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:05 smithi122 ceph-mon[27585]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-11T21:50:05.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:05 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:05 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:05 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:05 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:05 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:05 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:05 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:05 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.925 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:05 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:05 smithi002 ceph-mon[25764]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-11T21:50:05.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:05 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:05 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:05 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:05 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:05 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:05 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:05 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:05 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:05.929 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:05 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:06.041 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:05 smithi165 ceph-mon[27942]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-11T21:50:06.041 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:05 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:06.041 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:05 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:06.041 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:05 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:06.041 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:05 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:06.041 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:05 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:06.041 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:05 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:06.041 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:05 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:06.041 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:05 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:06.041 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:05 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:06.737 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:07.926 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:07 smithi122 ceph-mon[27585]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T21:50:07.926 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:07 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:07.926 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:07 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:07.926 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:07 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:07.926 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:07 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:07.926 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:07 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:07.926 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:07 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:07.926 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:07 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:08.038 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:07 smithi165 ceph-mon[27942]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T21:50:08.038 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:07 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:08.038 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:07 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:08.039 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:07 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:08.039 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:07 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:08.039 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:07 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:08.039 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:07 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:08.039 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:07 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:08.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:07 smithi002 ceph-mon[25764]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T21:50:08.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:07 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:08.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:07 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:08.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:07 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:08.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:07 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:08.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:07 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:08.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:07 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:08.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:07 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:09.288 INFO:teuthology.orchestra.run.smithi002.stdout:34359738390 2024-03-11T21:50:10.041 INFO:teuthology.orchestra.run.smithi002.stdout:77309411341 2024-03-11T21:50:10.041 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:09 smithi002 ceph-mon[25764]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T21:50:10.041 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:09 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/1402468635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:50:10.084 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:09 smithi165 ceph-mon[27942]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T21:50:10.084 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:09 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/1402468635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:50:10.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:09 smithi122 ceph-mon[27585]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T21:50:10.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:09 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/1402468635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:50:10.230 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738389 got 34359738390 for osd.0 2024-03-11T21:50:10.231 DEBUG:teuthology.parallel:result is None 2024-03-11T21:50:11.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:10 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/382807076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:50:11.094 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:10 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/382807076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:50:11.121 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411340 got 77309411341 for osd.2 2024-03-11T21:50:11.122 DEBUG:teuthology.parallel:result is None 2024-03-11T21:50:11.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:10 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/382807076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:50:11.891 INFO:teuthology.orchestra.run.smithi002.stdout:55834574867 2024-03-11T21:50:11.891 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:11 smithi002 ceph-mon[25764]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:50:12.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:11 smithi165 ceph-mon[27942]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:50:12.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:11 smithi122 ceph-mon[27585]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T21:50:12.513 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574865 got 55834574867 for osd.1 2024-03-11T21:50:12.514 DEBUG:teuthology.parallel:result is None 2024-03-11T21:50:12.514 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T21:50:12.514 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:50:13.034 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:12 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/4183936197' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:50:13.034 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:12 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:13.034 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:12 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:13.034 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:12 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:13.034 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:12 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:13.034 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:12 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:13.034 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:12 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:13.034 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:12 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:13.034 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:12 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:13.035 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:12 smithi002 ceph-mon[25764]: Deploying daemon alertmanager.a on smithi002 2024-03-11T21:50:13.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:12 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/4183936197' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:50:13.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:12 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:13.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:12 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:13.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:12 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:13.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:12 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:13.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:12 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:13.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:12 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:13.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:12 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:13.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:12 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:13.084 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:12 smithi165 ceph-mon[27942]: Deploying daemon alertmanager.a on smithi002 2024-03-11T21:50:13.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:12 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/4183936197' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:50:13.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:12 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:13.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:12 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:13.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:12 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:13.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:12 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:13.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:12 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:13.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:12 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:13.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:12 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:13.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:12 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:13.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:12 smithi122 ceph-mon[27585]: Deploying daemon alertmanager.a on smithi002 2024-03-11T21:50:13.189 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:14.328 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:14 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: ::ffff:172.21.15.122 - - [11/Mar/2024:21:50:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:14.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:14 smithi165 ceph-mon[27942]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:50:14.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:14 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:14.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:14 smithi002 ceph-mon[25764]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:50:14.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:14 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:14.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:14 smithi122 ceph-mon[27585]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:50:14.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:14 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:15.823 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:50:15.825 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-03-11T21:50:16.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:16 smithi165 ceph-mon[27942]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:50:16.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:16 smithi002 ceph-mon[25764]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:50:16.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:16 smithi122 ceph-mon[27585]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T21:50:16.990 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-03-11T21:50:14.715323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96616,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126552,"statfs":{"total":287972524032,"available":287873589248,"internally_reserved":0,"allocated":3158016,"data_stored":2100345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002827"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702944+0000","last_change":"2024-03-11T21:49:35.266574+0000","last_active":"2024-03-11T21:49:58.702944+0000","last_peered":"2024-03-11T21:49:58.702944+0000","last_clean":"2024-03-11T21:49:58.702944+0000","last_became_active":"2024-03-11T21:49:35.264787+0000","last_became_peered":"2024-03-11T21:49:35.264787+0000","last_unstale":"2024-03-11T21:49:58.702944+0000","last_undegraded":"2024-03-11T21:49:58.702944+0000","last_fullsized":"2024-03-11T21:49:58.702944+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:11:33.021725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702826+0000","last_change":"2024-03-11T21:49:33.258448+0000","last_active":"2024-03-11T21:49:58.702826+0000","last_peered":"2024-03-11T21:49:58.702826+0000","last_clean":"2024-03-11T21:49:58.702826+0000","last_became_active":"2024-03-11T21:49:33.257803+0000","last_became_peered":"2024-03-11T21:49:33.257803+0000","last_unstale":"2024-03-11T21:49:58.702826+0000","last_undegraded":"2024-03-11T21:49:58.702826+0000","last_fullsized":"2024-03-11T21:49:58.702826+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:46:57.467404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703960+0000","last_change":"2024-03-11T21:49:29.250428+0000","last_active":"2024-03-11T21:49:58.703960+0000","last_peered":"2024-03-11T21:49:58.703960+0000","last_clean":"2024-03-11T21:49:58.703960+0000","last_became_active":"2024-03-11T21:49:29.250107+0000","last_became_peered":"2024-03-11T21:49:29.250107+0000","last_unstale":"2024-03-11T21:49:58.703960+0000","last_undegraded":"2024-03-11T21:49:58.703960+0000","last_fullsized":"2024-03-11T21:49:58.703960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:15:27.588120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702728+0000","last_change":"2024-03-11T21:49:31.254145+0000","last_active":"2024-03-11T21:49:58.702728+0000","last_peered":"2024-03-11T21:49:58.702728+0000","last_clean":"2024-03-11T21:49:58.702728+0000","last_became_active":"2024-03-11T21:49:31.253848+0000","last_became_peered":"2024-03-11T21:49:31.253848+0000","last_unstale":"2024-03-11T21:49:58.702728+0000","last_undegraded":"2024-03-11T21:49:58.702728+0000","last_fullsized":"2024-03-11T21:49:58.702728+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:20:40.354767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700876+0000","last_change":"2024-03-11T21:49:35.267766+0000","last_active":"2024-03-11T21:49:58.700876+0000","last_peered":"2024-03-11T21:49:58.700876+0000","last_clean":"2024-03-11T21:49:58.700876+0000","last_became_active":"2024-03-11T21:49:35.266910+0000","last_became_peered":"2024-03-11T21:49:35.266910+0000","last_unstale":"2024-03-11T21:49:58.700876+0000","last_undegraded":"2024-03-11T21:49:58.700876+0000","last_fullsized":"2024-03-11T21:49:58.700876+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:22:37.209896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702352+0000","last_change":"2024-03-11T21:49:33.259921+0000","last_active":"2024-03-11T21:49:58.702352+0000","last_peered":"2024-03-11T21:49:58.702352+0000","last_clean":"2024-03-11T21:49:58.702352+0000","last_became_active":"2024-03-11T21:49:33.257940+0000","last_became_peered":"2024-03-11T21:49:33.257940+0000","last_unstale":"2024-03-11T21:49:58.702352+0000","last_undegraded":"2024-03-11T21:49:58.702352+0000","last_fullsized":"2024-03-11T21:49:58.702352+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:40:43.861759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703785+0000","last_change":"2024-03-11T21:49:29.251137+0000","last_active":"2024-03-11T21:49:58.703785+0000","last_peered":"2024-03-11T21:49:58.703785+0000","last_clean":"2024-03-11T21:49:58.703785+0000","last_became_active":"2024-03-11T21:49:29.250423+0000","last_became_peered":"2024-03-11T21:49:29.250423+0000","last_unstale":"2024-03-11T21:49:58.703785+0000","last_undegraded":"2024-03-11T21:49:58.703785+0000","last_fullsized":"2024-03-11T21:49:58.703785+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:40:18.849565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703754+0000","last_change":"2024-03-11T21:49:31.254702+0000","last_active":"2024-03-11T21:49:58.703754+0000","last_peered":"2024-03-11T21:49:58.703754+0000","last_clean":"2024-03-11T21:49:58.703754+0000","last_became_active":"2024-03-11T21:49:31.254167+0000","last_became_peered":"2024-03-11T21:49:31.254167+0000","last_unstale":"2024-03-11T21:49:58.703754+0000","last_undegraded":"2024-03-11T21:49:58.703754+0000","last_fullsized":"2024-03-11T21:49:58.703754+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:12:36.172908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703010+0000","last_change":"2024-03-11T21:49:35.265377+0000","last_active":"2024-03-11T21:49:58.703010+0000","last_peered":"2024-03-11T21:49:58.703010+0000","last_clean":"2024-03-11T21:49:58.703010+0000","last_became_active":"2024-03-11T21:49:35.265078+0000","last_became_peered":"2024-03-11T21:49:35.265078+0000","last_unstale":"2024-03-11T21:49:58.703010+0000","last_undegraded":"2024-03-11T21:49:58.703010+0000","last_fullsized":"2024-03-11T21:49:58.703010+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:21:58.784825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703845+0000","last_change":"2024-03-11T21:49:33.258161+0000","last_active":"2024-03-11T21:49:58.703845+0000","last_peered":"2024-03-11T21:49:58.703845+0000","last_clean":"2024-03-11T21:49:58.703845+0000","last_became_active":"2024-03-11T21:49:33.257786+0000","last_became_peered":"2024-03-11T21:49:33.257786+0000","last_unstale":"2024-03-11T21:49:58.703845+0000","last_undegraded":"2024-03-11T21:49:58.703845+0000","last_fullsized":"2024-03-11T21:49:58.703845+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:17:14.095458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705414+0000","last_change":"2024-03-11T21:49:29.254414+0000","last_active":"2024-03-11T21:49:58.705414+0000","last_peered":"2024-03-11T21:49:58.705414+0000","last_clean":"2024-03-11T21:49:58.705414+0000","last_became_active":"2024-03-11T21:49:29.252615+0000","last_became_peered":"2024-03-11T21:49:29.252615+0000","last_unstale":"2024-03-11T21:49:58.705414+0000","last_undegraded":"2024-03-11T21:49:58.705414+0000","last_fullsized":"2024-03-11T21:49:58.705414+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:31:58.317393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704899+0000","last_change":"2024-03-11T21:49:31.255253+0000","last_active":"2024-03-11T21:49:58.704899+0000","last_peered":"2024-03-11T21:49:58.704899+0000","last_clean":"2024-03-11T21:49:58.704899+0000","last_became_active":"2024-03-11T21:49:31.254165+0000","last_became_peered":"2024-03-11T21:49:31.254165+0000","last_unstale":"2024-03-11T21:49:58.704899+0000","last_undegraded":"2024-03-11T21:49:58.704899+0000","last_fullsized":"2024-03-11T21:49:58.704899+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:58:19.179754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702575+0000","last_change":"2024-03-11T21:49:35.266190+0000","last_active":"2024-03-11T21:49:58.702575+0000","last_peered":"2024-03-11T21:49:58.702575+0000","last_clean":"2024-03-11T21:49:58.702575+0000","last_became_active":"2024-03-11T21:49:35.265820+0000","last_became_peered":"2024-03-11T21:49:35.265820+0000","last_unstale":"2024-03-11T21:49:58.702575+0000","last_undegraded":"2024-03-11T21:49:58.702575+0000","last_fullsized":"2024-03-11T21:49:58.702575+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:50.869654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702330+0000","last_change":"2024-03-11T21:49:33.258267+0000","last_active":"2024-03-11T21:49:58.702330+0000","last_peered":"2024-03-11T21:49:58.702330+0000","last_clean":"2024-03-11T21:49:58.702330+0000","last_became_active":"2024-03-11T21:49:33.257841+0000","last_became_peered":"2024-03-11T21:49:33.257841+0000","last_unstale":"2024-03-11T21:49:58.702330+0000","last_undegraded":"2024-03-11T21:49:58.702330+0000","last_fullsized":"2024-03-11T21:49:58.702330+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:27:22.694373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704092+0000","last_change":"2024-03-11T21:49:29.253450+0000","last_active":"2024-03-11T21:49:58.704092+0000","last_peered":"2024-03-11T21:49:58.704092+0000","last_clean":"2024-03-11T21:49:58.704092+0000","last_became_active":"2024-03-11T21:49:29.253065+0000","last_became_peered":"2024-03-11T21:49:29.253065+0000","last_unstale":"2024-03-11T21:49:58.704092+0000","last_undegraded":"2024-03-11T21:49:58.704092+0000","last_fullsized":"2024-03-11T21:49:58.704092+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:37:08.139190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705297+0000","last_change":"2024-03-11T21:49:31.252312+0000","last_active":"2024-03-11T21:49:58.705297+0000","last_peered":"2024-03-11T21:49:58.705297+0000","last_clean":"2024-03-11T21:49:58.705297+0000","last_became_active":"2024-03-11T21:49:31.251928+0000","last_became_peered":"2024-03-11T21:49:31.251928+0000","last_unstale":"2024-03-11T21:49:58.705297+0000","last_undegraded":"2024-03-11T21:49:58.705297+0000","last_fullsized":"2024-03-11T21:49:58.705297+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:02:01.747036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703137+0000","last_change":"2024-03-11T21:49:35.266507+0000","last_active":"2024-03-11T21:49:58.703137+0000","last_peered":"2024-03-11T21:49:58.703137+0000","last_clean":"2024-03-11T21:49:58.703137+0000","last_became_active":"2024-03-11T21:49:35.266074+0000","last_became_peered":"2024-03-11T21:49:35.266074+0000","last_unstale":"2024-03-11T21:49:58.703137+0000","last_undegraded":"2024-03-11T21:49:58.703137+0000","last_fullsized":"2024-03-11T21:49:58.703137+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:06:27.011361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704712+0000","last_change":"2024-03-11T21:49:33.259967+0000","last_active":"2024-03-11T21:49:58.704712+0000","last_peered":"2024-03-11T21:49:58.704712+0000","last_clean":"2024-03-11T21:49:58.704712+0000","last_became_active":"2024-03-11T21:49:33.259511+0000","last_became_peered":"2024-03-11T21:49:33.259511+0000","last_unstale":"2024-03-11T21:49:58.704712+0000","last_undegraded":"2024-03-11T21:49:58.704712+0000","last_fullsized":"2024-03-11T21:49:58.704712+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:17:54.755379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704789+0000","last_change":"2024-03-11T21:49:29.253158+0000","last_active":"2024-03-11T21:49:58.704789+0000","last_peered":"2024-03-11T21:49:58.704789+0000","last_clean":"2024-03-11T21:49:58.704789+0000","last_became_active":"2024-03-11T21:49:29.252094+0000","last_became_peered":"2024-03-11T21:49:29.252094+0000","last_unstale":"2024-03-11T21:49:58.704789+0000","last_undegraded":"2024-03-11T21:49:58.704789+0000","last_fullsized":"2024-03-11T21:49:58.704789+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:30:37.823365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703688+0000","last_change":"2024-03-11T21:49:31.251983+0000","last_active":"2024-03-11T21:49:58.703688+0000","last_peered":"2024-03-11T21:49:58.703688+0000","last_clean":"2024-03-11T21:49:58.703688+0000","last_became_active":"2024-03-11T21:49:31.251356+0000","last_became_peered":"2024-03-11T21:49:31.251356+0000","last_unstale":"2024-03-11T21:49:58.703688+0000","last_undegraded":"2024-03-11T21:49:58.703688+0000","last_fullsized":"2024-03-11T21:49:58.703688+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:36:24.329449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702025+0000","last_change":"2024-03-11T21:49:35.267124+0000","last_active":"2024-03-11T21:49:58.702025+0000","last_peered":"2024-03-11T21:49:58.702025+0000","last_clean":"2024-03-11T21:49:58.702025+0000","last_became_active":"2024-03-11T21:49:35.266210+0000","last_became_peered":"2024-03-11T21:49:35.266210+0000","last_unstale":"2024-03-11T21:49:58.702025+0000","last_undegraded":"2024-03-11T21:49:58.702025+0000","last_fullsized":"2024-03-11T21:49:58.702025+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:42:54.996928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704000+0000","last_change":"2024-03-11T21:49:33.259271+0000","last_active":"2024-03-11T21:49:58.704000+0000","last_peered":"2024-03-11T21:49:58.704000+0000","last_clean":"2024-03-11T21:49:58.704000+0000","last_became_active":"2024-03-11T21:49:33.258689+0000","last_became_peered":"2024-03-11T21:49:33.258689+0000","last_unstale":"2024-03-11T21:49:58.704000+0000","last_undegraded":"2024-03-11T21:49:58.704000+0000","last_fullsized":"2024-03-11T21:49:58.704000+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:38:48.583083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702733+0000","last_change":"2024-03-11T21:49:29.251212+0000","last_active":"2024-03-11T21:49:58.702733+0000","last_peered":"2024-03-11T21:49:58.702733+0000","last_clean":"2024-03-11T21:49:58.702733+0000","last_became_active":"2024-03-11T21:49:29.250181+0000","last_became_peered":"2024-03-11T21:49:29.250181+0000","last_unstale":"2024-03-11T21:49:58.702733+0000","last_undegraded":"2024-03-11T21:49:58.702733+0000","last_fullsized":"2024-03-11T21:49:58.702733+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:00:11.889227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'5","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704589+0000","last_change":"2024-03-11T21:49:31.251994+0000","last_active":"2024-03-11T21:49:58.704589+0000","last_peered":"2024-03-11T21:49:58.704589+0000","last_clean":"2024-03-11T21:49:58.704589+0000","last_became_active":"2024-03-11T21:49:31.251639+0000","last_became_peered":"2024-03-11T21:49:31.251639+0000","last_unstale":"2024-03-11T21:49:58.704589+0000","last_undegraded":"2024-03-11T21:49:58.704589+0000","last_fullsized":"2024-03-11T21:49:58.704589+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:48.857302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702082+0000","last_change":"2024-03-11T21:49:35.266794+0000","last_active":"2024-03-11T21:49:58.702082+0000","last_peered":"2024-03-11T21:49:58.702082+0000","last_clean":"2024-03-11T21:49:58.702082+0000","last_became_active":"2024-03-11T21:49:35.265366+0000","last_became_peered":"2024-03-11T21:49:35.265366+0000","last_unstale":"2024-03-11T21:49:58.702082+0000","last_undegraded":"2024-03-11T21:49:58.702082+0000","last_fullsized":"2024-03-11T21:49:58.702082+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:39:15.017312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:04.305524+0000","last_change":"2024-03-11T21:49:33.260199+0000","last_active":"2024-03-11T21:50:04.305524+0000","last_peered":"2024-03-11T21:50:04.305524+0000","last_clean":"2024-03-11T21:50:04.305524+0000","last_became_active":"2024-03-11T21:49:33.259086+0000","last_became_peered":"2024-03-11T21:49:33.259086+0000","last_unstale":"2024-03-11T21:50:04.305524+0000","last_undegraded":"2024-03-11T21:50:04.305524+0000","last_fullsized":"2024-03-11T21:50:04.305524+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:21:46.569479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701157+0000","last_change":"2024-03-11T21:49:29.250365+0000","last_active":"2024-03-11T21:49:58.701157+0000","last_peered":"2024-03-11T21:49:58.701157+0000","last_clean":"2024-03-11T21:49:58.701157+0000","last_became_active":"2024-03-11T21:49:29.250013+0000","last_became_peered":"2024-03-11T21:49:29.250013+0000","last_unstale":"2024-03-11T21:49:58.701157+0000","last_undegraded":"2024-03-11T21:49:58.701157+0000","last_fullsized":"2024-03-11T21:49:58.701157+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:42:10.961047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700746+0000","last_change":"2024-03-11T21:49:31.251657+0000","last_active":"2024-03-11T21:49:58.700746+0000","last_peered":"2024-03-11T21:49:58.700746+0000","last_clean":"2024-03-11T21:49:58.700746+0000","last_became_active":"2024-03-11T21:49:31.251270+0000","last_became_peered":"2024-03-11T21:49:31.251270+0000","last_unstale":"2024-03-11T21:49:58.700746+0000","last_undegraded":"2024-03-11T21:49:58.700746+0000","last_fullsized":"2024-03-11T21:49:58.700746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:50:42.839109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704600+0000","last_change":"2024-03-11T21:49:35.267448+0000","last_active":"2024-03-11T21:49:58.704600+0000","last_peered":"2024-03-11T21:49:58.704600+0000","last_clean":"2024-03-11T21:49:58.704600+0000","last_became_active":"2024-03-11T21:49:35.266795+0000","last_became_peered":"2024-03-11T21:49:35.266795+0000","last_unstale":"2024-03-11T21:49:58.704600+0000","last_undegraded":"2024-03-11T21:49:58.704600+0000","last_fullsized":"2024-03-11T21:49:58.704600+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:13:51.922059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702266+0000","last_change":"2024-03-11T21:49:33.262434+0000","last_active":"2024-03-11T21:49:58.702266+0000","last_peered":"2024-03-11T21:49:58.702266+0000","last_clean":"2024-03-11T21:49:58.702266+0000","last_became_active":"2024-03-11T21:49:33.261983+0000","last_became_peered":"2024-03-11T21:49:33.261983+0000","last_unstale":"2024-03-11T21:49:58.702266+0000","last_undegraded":"2024-03-11T21:49:58.702266+0000","last_fullsized":"2024-03-11T21:49:58.702266+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:31:08.122586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701995+0000","last_change":"2024-03-11T21:49:29.252638+0000","last_active":"2024-03-11T21:49:58.701995+0000","last_peered":"2024-03-11T21:49:58.701995+0000","last_clean":"2024-03-11T21:49:58.701995+0000","last_became_active":"2024-03-11T21:49:29.252053+0000","last_became_peered":"2024-03-11T21:49:29.252053+0000","last_unstale":"2024-03-11T21:49:58.701995+0000","last_undegraded":"2024-03-11T21:49:58.701995+0000","last_fullsized":"2024-03-11T21:49:58.701995+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:33:16.238062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700713+0000","last_change":"2024-03-11T21:49:31.251741+0000","last_active":"2024-03-11T21:49:58.700713+0000","last_peered":"2024-03-11T21:49:58.700713+0000","last_clean":"2024-03-11T21:49:58.700713+0000","last_became_active":"2024-03-11T21:49:31.251444+0000","last_became_peered":"2024-03-11T21:49:31.251444+0000","last_unstale":"2024-03-11T21:49:58.700713+0000","last_undegraded":"2024-03-11T21:49:58.700713+0000","last_fullsized":"2024-03-11T21:49:58.700713+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:13:10.243341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702655+0000","last_change":"2024-03-11T21:49:35.266395+0000","last_active":"2024-03-11T21:49:58.702655+0000","last_peered":"2024-03-11T21:49:58.702655+0000","last_clean":"2024-03-11T21:49:58.702655+0000","last_became_active":"2024-03-11T21:49:35.265703+0000","last_became_peered":"2024-03-11T21:49:35.265703+0000","last_unstale":"2024-03-11T21:49:58.702655+0000","last_undegraded":"2024-03-11T21:49:58.702655+0000","last_fullsized":"2024-03-11T21:49:58.702655+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:28:28.143541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700591+0000","last_change":"2024-03-11T21:49:33.256796+0000","last_active":"2024-03-11T21:49:58.700591+0000","last_peered":"2024-03-11T21:49:58.700591+0000","last_clean":"2024-03-11T21:49:58.700591+0000","last_became_active":"2024-03-11T21:49:33.256236+0000","last_became_peered":"2024-03-11T21:49:33.256236+0000","last_unstale":"2024-03-11T21:49:58.700591+0000","last_undegraded":"2024-03-11T21:49:58.700591+0000","last_fullsized":"2024-03-11T21:49:58.700591+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:14:24.408240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705549+0000","last_change":"2024-03-11T21:49:29.253555+0000","last_active":"2024-03-11T21:49:58.705549+0000","last_peered":"2024-03-11T21:49:58.705549+0000","last_clean":"2024-03-11T21:49:58.705549+0000","last_became_active":"2024-03-11T21:49:29.253166+0000","last_became_peered":"2024-03-11T21:49:29.253166+0000","last_unstale":"2024-03-11T21:49:58.705549+0000","last_undegraded":"2024-03-11T21:49:58.705549+0000","last_fullsized":"2024-03-11T21:49:58.705549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:50:15.090986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704956+0000","last_change":"2024-03-11T21:49:31.255719+0000","last_active":"2024-03-11T21:49:58.704956+0000","last_peered":"2024-03-11T21:49:58.704956+0000","last_clean":"2024-03-11T21:49:58.704956+0000","last_became_active":"2024-03-11T21:49:31.254802+0000","last_became_peered":"2024-03-11T21:49:31.254802+0000","last_unstale":"2024-03-11T21:49:58.704956+0000","last_undegraded":"2024-03-11T21:49:58.704956+0000","last_fullsized":"2024-03-11T21:49:58.704956+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:13:30.795360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702317+0000","last_change":"2024-03-11T21:49:35.264816+0000","last_active":"2024-03-11T21:49:58.702317+0000","last_peered":"2024-03-11T21:49:58.702317+0000","last_clean":"2024-03-11T21:49:58.702317+0000","last_became_active":"2024-03-11T21:49:35.263949+0000","last_became_peered":"2024-03-11T21:49:35.263949+0000","last_unstale":"2024-03-11T21:49:58.702317+0000","last_undegraded":"2024-03-11T21:49:58.702317+0000","last_fullsized":"2024-03-11T21:49:58.702317+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:15:15.392421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700948+0000","last_change":"2024-03-11T21:49:33.257120+0000","last_active":"2024-03-11T21:49:58.700948+0000","last_peered":"2024-03-11T21:49:58.700948+0000","last_clean":"2024-03-11T21:49:58.700948+0000","last_became_active":"2024-03-11T21:49:33.256827+0000","last_became_peered":"2024-03-11T21:49:33.256827+0000","last_unstale":"2024-03-11T21:49:58.700948+0000","last_undegraded":"2024-03-11T21:49:58.700948+0000","last_fullsized":"2024-03-11T21:49:58.700948+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:53:15.292904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704639+0000","last_change":"2024-03-11T21:49:29.253879+0000","last_active":"2024-03-11T21:49:58.704639+0000","last_peered":"2024-03-11T21:49:58.704639+0000","last_clean":"2024-03-11T21:49:58.704639+0000","last_became_active":"2024-03-11T21:49:29.253502+0000","last_became_peered":"2024-03-11T21:49:29.253502+0000","last_unstale":"2024-03-11T21:49:58.704639+0000","last_undegraded":"2024-03-11T21:49:58.704639+0000","last_fullsized":"2024-03-11T21:49:58.704639+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:54:11.900918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'6","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704176+0000","last_change":"2024-03-11T21:49:31.254462+0000","last_active":"2024-03-11T21:49:58.704176+0000","last_peered":"2024-03-11T21:49:58.704176+0000","last_clean":"2024-03-11T21:49:58.704176+0000","last_became_active":"2024-03-11T21:49:31.254114+0000","last_became_peered":"2024-03-11T21:49:31.254114+0000","last_unstale":"2024-03-11T21:49:58.704176+0000","last_undegraded":"2024-03-11T21:49:58.704176+0000","last_fullsized":"2024-03-11T21:49:58.704176+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:05:18.897284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703166+0000","last_change":"2024-03-11T21:49:35.266310+0000","last_active":"2024-03-11T21:49:58.703166+0000","last_peered":"2024-03-11T21:49:58.703166+0000","last_clean":"2024-03-11T21:49:58.703166+0000","last_became_active":"2024-03-11T21:49:35.265833+0000","last_became_peered":"2024-03-11T21:49:35.265833+0000","last_unstale":"2024-03-11T21:49:58.703166+0000","last_undegraded":"2024-03-11T21:49:58.703166+0000","last_fullsized":"2024-03-11T21:49:58.703166+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:23:08.633779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703954+0000","last_change":"2024-03-11T21:49:33.260132+0000","last_active":"2024-03-11T21:49:58.703954+0000","last_peered":"2024-03-11T21:49:58.703954+0000","last_clean":"2024-03-11T21:49:58.703954+0000","last_became_active":"2024-03-11T21:49:33.259118+0000","last_became_peered":"2024-03-11T21:49:33.259118+0000","last_unstale":"2024-03-11T21:49:58.703954+0000","last_undegraded":"2024-03-11T21:49:58.703954+0000","last_fullsized":"2024-03-11T21:49:58.703954+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:02.863183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705276+0000","last_change":"2024-03-11T21:49:29.252975+0000","last_active":"2024-03-11T21:49:58.705276+0000","last_peered":"2024-03-11T21:49:58.705276+0000","last_clean":"2024-03-11T21:49:58.705276+0000","last_became_active":"2024-03-11T21:49:29.252236+0000","last_became_peered":"2024-03-11T21:49:29.252236+0000","last_unstale":"2024-03-11T21:49:58.705276+0000","last_undegraded":"2024-03-11T21:49:58.705276+0000","last_fullsized":"2024-03-11T21:49:58.705276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:11:32.206643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704543+0000","last_change":"2024-03-11T21:49:31.255351+0000","last_active":"2024-03-11T21:49:58.704543+0000","last_peered":"2024-03-11T21:49:58.704543+0000","last_clean":"2024-03-11T21:49:58.704543+0000","last_became_active":"2024-03-11T21:49:31.254904+0000","last_became_peered":"2024-03-11T21:49:31.254904+0000","last_unstale":"2024-03-11T21:49:58.704543+0000","last_undegraded":"2024-03-11T21:49:58.704543+0000","last_fullsized":"2024-03-11T21:49:58.704543+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:13:57.471498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703346+0000","last_change":"2024-03-11T21:49:35.266079+0000","last_active":"2024-03-11T21:49:58.703346+0000","last_peered":"2024-03-11T21:49:58.703346+0000","last_clean":"2024-03-11T21:49:58.703346+0000","last_became_active":"2024-03-11T21:49:35.265750+0000","last_became_peered":"2024-03-11T21:49:35.265750+0000","last_unstale":"2024-03-11T21:49:58.703346+0000","last_undegraded":"2024-03-11T21:49:58.703346+0000","last_fullsized":"2024-03-11T21:49:58.703346+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:08:02.225852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701269+0000","last_change":"2024-03-11T21:49:33.257157+0000","last_active":"2024-03-11T21:49:58.701269+0000","last_peered":"2024-03-11T21:49:58.701269+0000","last_clean":"2024-03-11T21:49:58.701269+0000","last_became_active":"2024-03-11T21:49:33.256678+0000","last_became_peered":"2024-03-11T21:49:33.256678+0000","last_unstale":"2024-03-11T21:49:58.701269+0000","last_undegraded":"2024-03-11T21:49:58.701269+0000","last_fullsized":"2024-03-11T21:49:58.701269+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:13:41.924302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703449+0000","last_change":"2024-03-11T21:49:29.254065+0000","last_active":"2024-03-11T21:49:58.703449+0000","last_peered":"2024-03-11T21:49:58.703449+0000","last_clean":"2024-03-11T21:49:58.703449+0000","last_became_active":"2024-03-11T21:49:29.253496+0000","last_became_peered":"2024-03-11T21:49:29.253496+0000","last_unstale":"2024-03-11T21:49:58.703449+0000","last_undegraded":"2024-03-11T21:49:58.703449+0000","last_fullsized":"2024-03-11T21:49:58.703449+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:58:47.368433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705009+0000","last_change":"2024-03-11T21:49:31.251775+0000","last_active":"2024-03-11T21:49:58.705009+0000","last_peered":"2024-03-11T21:49:58.705009+0000","last_clean":"2024-03-11T21:49:58.705009+0000","last_became_active":"2024-03-11T21:49:31.251467+0000","last_became_peered":"2024-03-11T21:49:31.251467+0000","last_unstale":"2024-03-11T21:49:58.705009+0000","last_undegraded":"2024-03-11T21:49:58.705009+0000","last_fullsized":"2024-03-11T21:49:58.705009+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:53:17.227075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702268+0000","last_change":"2024-03-11T21:49:35.266496+0000","last_active":"2024-03-11T21:49:58.702268+0000","last_peered":"2024-03-11T21:49:58.702268+0000","last_clean":"2024-03-11T21:49:58.702268+0000","last_became_active":"2024-03-11T21:49:35.266167+0000","last_became_peered":"2024-03-11T21:49:35.266167+0000","last_unstale":"2024-03-11T21:49:58.702268+0000","last_undegraded":"2024-03-11T21:49:58.702268+0000","last_fullsized":"2024-03-11T21:49:58.702268+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:18:14.458344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:04.305544+0000","last_change":"2024-03-11T21:49:33.258873+0000","last_active":"2024-03-11T21:50:04.305544+0000","last_peered":"2024-03-11T21:50:04.305544+0000","last_clean":"2024-03-11T21:50:04.305544+0000","last_became_active":"2024-03-11T21:49:33.258554+0000","last_became_peered":"2024-03-11T21:49:33.258554+0000","last_unstale":"2024-03-11T21:50:04.305544+0000","last_undegraded":"2024-03-11T21:50:04.305544+0000","last_fullsized":"2024-03-11T21:50:04.305544+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:30:23.282288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704883+0000","last_change":"2024-03-11T21:49:29.254717+0000","last_active":"2024-03-11T21:49:58.704883+0000","last_peered":"2024-03-11T21:49:58.704883+0000","last_clean":"2024-03-11T21:49:58.704883+0000","last_became_active":"2024-03-11T21:49:29.253408+0000","last_became_peered":"2024-03-11T21:49:29.253408+0000","last_unstale":"2024-03-11T21:49:58.704883+0000","last_undegraded":"2024-03-11T21:49:58.704883+0000","last_fullsized":"2024-03-11T21:49:58.704883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:47:36.506975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701496+0000","last_change":"2024-03-11T21:49:31.252136+0000","last_active":"2024-03-11T21:49:58.701496+0000","last_peered":"2024-03-11T21:49:58.701496+0000","last_clean":"2024-03-11T21:49:58.701496+0000","last_became_active":"2024-03-11T21:49:31.251829+0000","last_became_peered":"2024-03-11T21:49:31.251829+0000","last_unstale":"2024-03-11T21:49:58.701496+0000","last_undegraded":"2024-03-11T21:49:58.701496+0000","last_fullsized":"2024-03-11T21:49:58.701496+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:11:35.422733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702953+0000","last_change":"2024-03-11T21:49:35.264377+0000","last_active":"2024-03-11T21:49:58.702953+0000","last_peered":"2024-03-11T21:49:58.702953+0000","last_clean":"2024-03-11T21:49:58.702953+0000","last_became_active":"2024-03-11T21:49:35.263974+0000","last_became_peered":"2024-03-11T21:49:35.263974+0000","last_unstale":"2024-03-11T21:49:58.702953+0000","last_undegraded":"2024-03-11T21:49:58.702953+0000","last_fullsized":"2024-03-11T21:49:58.702953+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:18:18.828987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:04.305640+0000","last_change":"2024-03-11T21:49:33.258616+0000","last_active":"2024-03-11T21:50:04.305640+0000","last_peered":"2024-03-11T21:50:04.305640+0000","last_clean":"2024-03-11T21:50:04.305640+0000","last_became_active":"2024-03-11T21:49:33.258280+0000","last_became_peered":"2024-03-11T21:49:33.258280+0000","last_unstale":"2024-03-11T21:50:04.305640+0000","last_undegraded":"2024-03-11T21:50:04.305640+0000","last_fullsized":"2024-03-11T21:50:04.305640+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:15:08.416440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703363+0000","last_change":"2024-03-11T21:49:29.254381+0000","last_active":"2024-03-11T21:49:58.703363+0000","last_peered":"2024-03-11T21:49:58.703363+0000","last_clean":"2024-03-11T21:49:58.703363+0000","last_became_active":"2024-03-11T21:49:29.253076+0000","last_became_peered":"2024-03-11T21:49:29.253076+0000","last_unstale":"2024-03-11T21:49:58.703363+0000","last_undegraded":"2024-03-11T21:49:58.703363+0000","last_fullsized":"2024-03-11T21:49:58.703363+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:17.066044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704810+0000","last_change":"2024-03-11T21:49:31.254237+0000","last_active":"2024-03-11T21:49:58.704810+0000","last_peered":"2024-03-11T21:49:58.704810+0000","last_clean":"2024-03-11T21:49:58.704810+0000","last_became_active":"2024-03-11T21:49:31.253830+0000","last_became_peered":"2024-03-11T21:49:31.253830+0000","last_unstale":"2024-03-11T21:49:58.704810+0000","last_undegraded":"2024-03-11T21:49:58.704810+0000","last_fullsized":"2024-03-11T21:49:58.704810+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:33:30.565887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703889+0000","last_change":"2024-03-11T21:49:35.264549+0000","last_active":"2024-03-11T21:49:58.703889+0000","last_peered":"2024-03-11T21:49:58.703889+0000","last_clean":"2024-03-11T21:49:58.703889+0000","last_became_active":"2024-03-11T21:49:35.264106+0000","last_became_peered":"2024-03-11T21:49:35.264106+0000","last_unstale":"2024-03-11T21:49:58.703889+0000","last_undegraded":"2024-03-11T21:49:58.703889+0000","last_fullsized":"2024-03-11T21:49:58.703889+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:40:58.738439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702669+0000","last_change":"2024-03-11T21:49:33.263001+0000","last_active":"2024-03-11T21:49:58.702669+0000","last_peered":"2024-03-11T21:49:58.702669+0000","last_clean":"2024-03-11T21:49:58.702669+0000","last_became_active":"2024-03-11T21:49:33.261811+0000","last_became_peered":"2024-03-11T21:49:33.261811+0000","last_unstale":"2024-03-11T21:49:58.702669+0000","last_undegraded":"2024-03-11T21:49:58.702669+0000","last_fullsized":"2024-03-11T21:49:58.702669+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:58.974435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702455+0000","last_change":"2024-03-11T21:49:29.253790+0000","last_active":"2024-03-11T21:49:58.702455+0000","last_peered":"2024-03-11T21:49:58.702455+0000","last_clean":"2024-03-11T21:49:58.702455+0000","last_became_active":"2024-03-11T21:49:29.253413+0000","last_became_peered":"2024-03-11T21:49:29.253413+0000","last_unstale":"2024-03-11T21:49:58.702455+0000","last_undegraded":"2024-03-11T21:49:58.702455+0000","last_fullsized":"2024-03-11T21:49:58.702455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:59:28.328961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702039+0000","last_change":"2024-03-11T21:49:31.254536+0000","last_active":"2024-03-11T21:49:58.702039+0000","last_peered":"2024-03-11T21:49:58.702039+0000","last_clean":"2024-03-11T21:49:58.702039+0000","last_became_active":"2024-03-11T21:49:31.253975+0000","last_became_peered":"2024-03-11T21:49:31.253975+0000","last_unstale":"2024-03-11T21:49:58.702039+0000","last_undegraded":"2024-03-11T21:49:58.702039+0000","last_fullsized":"2024-03-11T21:49:58.702039+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:50:23.411882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702390+0000","last_change":"2024-03-11T21:49:35.266623+0000","last_active":"2024-03-11T21:49:58.702390+0000","last_peered":"2024-03-11T21:49:58.702390+0000","last_clean":"2024-03-11T21:49:58.702390+0000","last_became_active":"2024-03-11T21:49:35.266263+0000","last_became_peered":"2024-03-11T21:49:35.266263+0000","last_unstale":"2024-03-11T21:49:58.702390+0000","last_undegraded":"2024-03-11T21:49:58.702390+0000","last_fullsized":"2024-03-11T21:49:58.702390+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:57:21.861934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702381+0000","last_change":"2024-03-11T21:49:33.262833+0000","last_active":"2024-03-11T21:49:58.702381+0000","last_peered":"2024-03-11T21:49:58.702381+0000","last_clean":"2024-03-11T21:49:58.702381+0000","last_became_active":"2024-03-11T21:49:33.262539+0000","last_became_peered":"2024-03-11T21:49:33.262539+0000","last_unstale":"2024-03-11T21:49:58.702381+0000","last_undegraded":"2024-03-11T21:49:58.702381+0000","last_fullsized":"2024-03-11T21:49:58.702381+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:51:04.157416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702066+0000","last_change":"2024-03-11T21:49:29.254501+0000","last_active":"2024-03-11T21:49:58.702066+0000","last_peered":"2024-03-11T21:49:58.702066+0000","last_clean":"2024-03-11T21:49:58.702066+0000","last_became_active":"2024-03-11T21:49:29.253334+0000","last_became_peered":"2024-03-11T21:49:29.253334+0000","last_unstale":"2024-03-11T21:49:58.702066+0000","last_undegraded":"2024-03-11T21:49:58.702066+0000","last_fullsized":"2024-03-11T21:49:58.702066+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:09:41.126405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703258+0000","last_change":"2024-03-11T21:49:31.251783+0000","last_active":"2024-03-11T21:49:58.703258+0000","last_peered":"2024-03-11T21:49:58.703258+0000","last_clean":"2024-03-11T21:49:58.703258+0000","last_became_active":"2024-03-11T21:49:31.251316+0000","last_became_peered":"2024-03-11T21:49:31.251316+0000","last_unstale":"2024-03-11T21:49:58.703258+0000","last_undegraded":"2024-03-11T21:49:58.703258+0000","last_fullsized":"2024-03-11T21:49:58.703258+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:10.579484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702308+0000","last_change":"2024-03-11T21:49:35.266224+0000","last_active":"2024-03-11T21:49:58.702308+0000","last_peered":"2024-03-11T21:49:58.702308+0000","last_clean":"2024-03-11T21:49:58.702308+0000","last_became_active":"2024-03-11T21:49:35.265890+0000","last_became_peered":"2024-03-11T21:49:35.265890+0000","last_unstale":"2024-03-11T21:49:58.702308+0000","last_undegraded":"2024-03-11T21:49:58.702308+0000","last_fullsized":"2024-03-11T21:49:58.702308+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:16:49.683222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704366+0000","last_change":"2024-03-11T21:49:29.252605+0000","last_active":"2024-03-11T21:49:58.704366+0000","last_peered":"2024-03-11T21:49:58.704366+0000","last_clean":"2024-03-11T21:49:58.704366+0000","last_became_active":"2024-03-11T21:49:29.252054+0000","last_became_peered":"2024-03-11T21:49:29.252054+0000","last_unstale":"2024-03-11T21:49:58.704366+0000","last_undegraded":"2024-03-11T21:49:58.704366+0000","last_fullsized":"2024-03-11T21:49:58.704366+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:04:42.686346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704492+0000","last_change":"2024-03-11T21:49:31.253985+0000","last_active":"2024-03-11T21:49:58.704492+0000","last_peered":"2024-03-11T21:49:58.704492+0000","last_clean":"2024-03-11T21:49:58.704492+0000","last_became_active":"2024-03-11T21:49:31.253665+0000","last_became_peered":"2024-03-11T21:49:31.253665+0000","last_unstale":"2024-03-11T21:49:58.704492+0000","last_undegraded":"2024-03-11T21:49:58.704492+0000","last_fullsized":"2024-03-11T21:49:58.704492+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:20:12.330839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:04.305620+0000","last_change":"2024-03-11T21:49:33.258937+0000","last_active":"2024-03-11T21:50:04.305620+0000","last_peered":"2024-03-11T21:50:04.305620+0000","last_clean":"2024-03-11T21:50:04.305620+0000","last_became_active":"2024-03-11T21:49:33.257996+0000","last_became_peered":"2024-03-11T21:49:33.257996+0000","last_unstale":"2024-03-11T21:50:04.305620+0000","last_undegraded":"2024-03-11T21:50:04.305620+0000","last_fullsized":"2024-03-11T21:50:04.305620+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:34:51.743830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700858+0000","last_change":"2024-03-11T21:49:35.265004+0000","last_active":"2024-03-11T21:49:58.700858+0000","last_peered":"2024-03-11T21:49:58.700858+0000","last_clean":"2024-03-11T21:49:58.700858+0000","last_became_active":"2024-03-11T21:49:35.264509+0000","last_became_peered":"2024-03-11T21:49:35.264509+0000","last_unstale":"2024-03-11T21:49:58.700858+0000","last_undegraded":"2024-03-11T21:49:58.700858+0000","last_fullsized":"2024-03-11T21:49:58.700858+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:46:13.626612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702551+0000","last_change":"2024-03-11T21:49:29.250000+0000","last_active":"2024-03-11T21:49:58.702551+0000","last_peered":"2024-03-11T21:49:58.702551+0000","last_clean":"2024-03-11T21:49:58.702551+0000","last_became_active":"2024-03-11T21:49:29.249114+0000","last_became_peered":"2024-03-11T21:49:29.249114+0000","last_unstale":"2024-03-11T21:49:58.702551+0000","last_undegraded":"2024-03-11T21:49:58.702551+0000","last_fullsized":"2024-03-11T21:49:58.702551+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:16:31.002441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:04.305664+0000","last_change":"2024-03-11T21:49:31.254374+0000","last_active":"2024-03-11T21:50:04.305664+0000","last_peered":"2024-03-11T21:50:04.305664+0000","last_clean":"2024-03-11T21:50:04.305664+0000","last_became_active":"2024-03-11T21:49:31.253872+0000","last_became_peered":"2024-03-11T21:49:31.253872+0000","last_unstale":"2024-03-11T21:50:04.305664+0000","last_undegraded":"2024-03-11T21:50:04.305664+0000","last_fullsized":"2024-03-11T21:50:04.305664+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:57:49.568838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703206+0000","last_change":"2024-03-11T21:49:33.259579+0000","last_active":"2024-03-11T21:49:58.703206+0000","last_peered":"2024-03-11T21:49:58.703206+0000","last_clean":"2024-03-11T21:49:58.703206+0000","last_became_active":"2024-03-11T21:49:33.258743+0000","last_became_peered":"2024-03-11T21:49:33.258743+0000","last_unstale":"2024-03-11T21:49:58.703206+0000","last_undegraded":"2024-03-11T21:49:58.703206+0000","last_fullsized":"2024-03-11T21:49:58.703206+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:43:13.877407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701610+0000","last_change":"2024-03-11T21:49:35.268647+0000","last_active":"2024-03-11T21:49:58.701610+0000","last_peered":"2024-03-11T21:49:58.701610+0000","last_clean":"2024-03-11T21:49:58.701610+0000","last_became_active":"2024-03-11T21:49:35.268328+0000","last_became_peered":"2024-03-11T21:49:35.268328+0000","last_unstale":"2024-03-11T21:49:58.701610+0000","last_undegraded":"2024-03-11T21:49:58.701610+0000","last_fullsized":"2024-03-11T21:49:58.701610+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:36:22.813279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703045+0000","last_change":"2024-03-11T21:49:29.251362+0000","last_active":"2024-03-11T21:49:58.703045+0000","last_peered":"2024-03-11T21:49:58.703045+0000","last_clean":"2024-03-11T21:49:58.703045+0000","last_became_active":"2024-03-11T21:49:29.250261+0000","last_became_peered":"2024-03-11T21:49:29.250261+0000","last_unstale":"2024-03-11T21:49:58.703045+0000","last_undegraded":"2024-03-11T21:49:58.703045+0000","last_fullsized":"2024-03-11T21:49:58.703045+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:12:59.338161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'19","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704963+0000","last_change":"2024-03-11T21:49:31.256262+0000","last_active":"2024-03-11T21:49:58.704963+0000","last_peered":"2024-03-11T21:49:58.704963+0000","last_clean":"2024-03-11T21:49:58.704963+0000","last_became_active":"2024-03-11T21:49:31.255764+0000","last_became_peered":"2024-03-11T21:49:31.255764+0000","last_unstale":"2024-03-11T21:49:58.704963+0000","last_undegraded":"2024-03-11T21:49:58.704963+0000","last_fullsized":"2024-03-11T21:49:58.704963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:17:36.847530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702318+0000","last_change":"2024-03-11T21:49:33.263685+0000","last_active":"2024-03-11T21:49:58.702318+0000","last_peered":"2024-03-11T21:49:58.702318+0000","last_clean":"2024-03-11T21:49:58.702318+0000","last_became_active":"2024-03-11T21:49:33.263404+0000","last_became_peered":"2024-03-11T21:49:33.263404+0000","last_unstale":"2024-03-11T21:49:58.702318+0000","last_undegraded":"2024-03-11T21:49:58.702318+0000","last_fullsized":"2024-03-11T21:49:58.702318+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:29:55.523250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703889+0000","last_change":"2024-03-11T21:49:35.268160+0000","last_active":"2024-03-11T21:49:58.703889+0000","last_peered":"2024-03-11T21:49:58.703889+0000","last_clean":"2024-03-11T21:49:58.703889+0000","last_became_active":"2024-03-11T21:49:35.267813+0000","last_became_peered":"2024-03-11T21:49:35.267813+0000","last_unstale":"2024-03-11T21:49:58.703889+0000","last_undegraded":"2024-03-11T21:49:58.703889+0000","last_fullsized":"2024-03-11T21:49:58.703889+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:19:59.956587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'18","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701871+0000","last_change":"2024-03-11T21:49:31.253876+0000","last_active":"2024-03-11T21:49:58.701871+0000","last_peered":"2024-03-11T21:49:58.701871+0000","last_clean":"2024-03-11T21:49:58.701871+0000","last_became_active":"2024-03-11T21:49:31.253542+0000","last_became_peered":"2024-03-11T21:49:31.253542+0000","last_unstale":"2024-03-11T21:49:58.701871+0000","last_undegraded":"2024-03-11T21:49:58.701871+0000","last_fullsized":"2024-03-11T21:49:58.701871+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:01:58.766660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700853+0000","last_change":"2024-03-11T21:49:29.250112+0000","last_active":"2024-03-11T21:49:58.700853+0000","last_peered":"2024-03-11T21:49:58.700853+0000","last_clean":"2024-03-11T21:49:58.700853+0000","last_became_active":"2024-03-11T21:49:29.249303+0000","last_became_peered":"2024-03-11T21:49:29.249303+0000","last_unstale":"2024-03-11T21:49:58.700853+0000","last_undegraded":"2024-03-11T21:49:58.700853+0000","last_fullsized":"2024-03-11T21:49:58.700853+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:39:04.972663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704108+0000","last_change":"2024-03-11T21:49:33.264210+0000","last_active":"2024-03-11T21:49:58.704108+0000","last_peered":"2024-03-11T21:49:58.704108+0000","last_clean":"2024-03-11T21:49:58.704108+0000","last_became_active":"2024-03-11T21:49:33.263904+0000","last_became_peered":"2024-03-11T21:49:33.263904+0000","last_unstale":"2024-03-11T21:49:58.704108+0000","last_undegraded":"2024-03-11T21:49:58.704108+0000","last_fullsized":"2024-03-11T21:49:58.704108+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:42.991166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703232+0000","last_change":"2024-03-11T21:49:35.269428+0000","last_active":"2024-03-11T21:49:58.703232+0000","last_peered":"2024-03-11T21:49:58.703232+0000","last_clean":"2024-03-11T21:49:58.703232+0000","last_became_active":"2024-03-11T21:49:35.269045+0000","last_became_peered":"2024-03-11T21:49:35.269045+0000","last_unstale":"2024-03-11T21:49:58.703232+0000","last_undegraded":"2024-03-11T21:49:58.703232+0000","last_fullsized":"2024-03-11T21:49:58.703232+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:21:55.158165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701686+0000","last_change":"2024-03-11T21:49:31.252365+0000","last_active":"2024-03-11T21:49:58.701686+0000","last_peered":"2024-03-11T21:49:58.701686+0000","last_clean":"2024-03-11T21:49:58.701686+0000","last_became_active":"2024-03-11T21:49:31.251505+0000","last_became_peered":"2024-03-11T21:49:31.251505+0000","last_unstale":"2024-03-11T21:49:58.701686+0000","last_undegraded":"2024-03-11T21:49:58.701686+0000","last_fullsized":"2024-03-11T21:49:58.701686+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:10:28.791536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704762+0000","last_change":"2024-03-11T21:49:29.252578+0000","last_active":"2024-03-11T21:49:58.704762+0000","last_peered":"2024-03-11T21:49:58.704762+0000","last_clean":"2024-03-11T21:49:58.704762+0000","last_became_active":"2024-03-11T21:49:29.251806+0000","last_became_peered":"2024-03-11T21:49:29.251806+0000","last_unstale":"2024-03-11T21:49:58.704762+0000","last_undegraded":"2024-03-11T21:49:58.704762+0000","last_fullsized":"2024-03-11T21:49:58.704762+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:53:15.456530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:00.807194+0000","last_change":"2024-03-11T21:49:17.908555+0000","last_active":"2024-03-11T21:50:00.807194+0000","last_peered":"2024-03-11T21:50:00.807194+0000","last_clean":"2024-03-11T21:50:00.807194+0000","last_became_active":"2024-03-11T21:49:17.908245+0000","last_became_peered":"2024-03-11T21:49:17.908245+0000","last_unstale":"2024-03-11T21:50:00.807194+0000","last_undegraded":"2024-03-11T21:50:00.807194+0000","last_fullsized":"2024-03-11T21:50:00.807194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:16.898621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:16.898621+0000","last_clean_scrub_stamp":"2024-03-11T21:49:16.898621+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:22:46.435509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703385+0000","last_change":"2024-03-11T21:49:33.259270+0000","last_active":"2024-03-11T21:49:58.703385+0000","last_peered":"2024-03-11T21:49:58.703385+0000","last_clean":"2024-03-11T21:49:58.703385+0000","last_became_active":"2024-03-11T21:49:33.258951+0000","last_became_peered":"2024-03-11T21:49:33.258951+0000","last_unstale":"2024-03-11T21:49:58.703385+0000","last_undegraded":"2024-03-11T21:49:58.703385+0000","last_fullsized":"2024-03-11T21:49:58.703385+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:34.972825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705392+0000","last_change":"2024-03-11T21:49:35.267216+0000","last_active":"2024-03-11T21:49:58.705392+0000","last_peered":"2024-03-11T21:49:58.705392+0000","last_clean":"2024-03-11T21:49:58.705392+0000","last_became_active":"2024-03-11T21:49:35.266225+0000","last_became_peered":"2024-03-11T21:49:35.266225+0000","last_unstale":"2024-03-11T21:49:58.705392+0000","last_undegraded":"2024-03-11T21:49:58.705392+0000","last_fullsized":"2024-03-11T21:49:58.705392+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:51:42.373614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'14","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701163+0000","last_change":"2024-03-11T21:49:31.251272+0000","last_active":"2024-03-11T21:49:58.701163+0000","last_peered":"2024-03-11T21:49:58.701163+0000","last_clean":"2024-03-11T21:49:58.701163+0000","last_became_active":"2024-03-11T21:49:31.250969+0000","last_became_peered":"2024-03-11T21:49:31.250969+0000","last_unstale":"2024-03-11T21:49:58.701163+0000","last_undegraded":"2024-03-11T21:49:58.701163+0000","last_fullsized":"2024-03-11T21:49:58.701163+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:53:14.801198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702137+0000","last_change":"2024-03-11T21:49:29.253080+0000","last_active":"2024-03-11T21:49:58.702137+0000","last_peered":"2024-03-11T21:49:58.702137+0000","last_clean":"2024-03-11T21:49:58.702137+0000","last_became_active":"2024-03-11T21:49:29.252371+0000","last_became_peered":"2024-03-11T21:49:29.252371+0000","last_unstale":"2024-03-11T21:49:58.702137+0000","last_undegraded":"2024-03-11T21:49:58.702137+0000","last_fullsized":"2024-03-11T21:49:58.702137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:25.249133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703764+0000","last_change":"2024-03-11T21:49:33.258715+0000","last_active":"2024-03-11T21:49:58.703764+0000","last_peered":"2024-03-11T21:49:58.703764+0000","last_clean":"2024-03-11T21:49:58.703764+0000","last_became_active":"2024-03-11T21:49:33.258402+0000","last_became_peered":"2024-03-11T21:49:33.258402+0000","last_unstale":"2024-03-11T21:49:58.703764+0000","last_undegraded":"2024-03-11T21:49:58.703764+0000","last_fullsized":"2024-03-11T21:49:58.703764+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:04:29.126653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702684+0000","last_change":"2024-03-11T21:49:35.266635+0000","last_active":"2024-03-11T21:49:58.702684+0000","last_peered":"2024-03-11T21:49:58.702684+0000","last_clean":"2024-03-11T21:49:58.702684+0000","last_became_active":"2024-03-11T21:49:35.265618+0000","last_became_peered":"2024-03-11T21:49:35.265618+0000","last_unstale":"2024-03-11T21:49:58.702684+0000","last_undegraded":"2024-03-11T21:49:58.702684+0000","last_fullsized":"2024-03-11T21:49:58.702684+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:12:21.022976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705195+0000","last_change":"2024-03-11T21:49:29.254818+0000","last_active":"2024-03-11T21:49:58.705195+0000","last_peered":"2024-03-11T21:49:58.705195+0000","last_clean":"2024-03-11T21:49:58.705195+0000","last_became_active":"2024-03-11T21:49:29.253609+0000","last_became_peered":"2024-03-11T21:49:29.253609+0000","last_unstale":"2024-03-11T21:49:58.705195+0000","last_undegraded":"2024-03-11T21:49:58.705195+0000","last_fullsized":"2024-03-11T21:49:58.705195+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:26:41.232304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'28","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:09.306576+0000","last_change":"2024-03-11T21:49:31.253200+0000","last_active":"2024-03-11T21:50:09.306576+0000","last_peered":"2024-03-11T21:50:09.306576+0000","last_clean":"2024-03-11T21:50:09.306576+0000","last_became_active":"2024-03-11T21:49:31.252914+0000","last_became_peered":"2024-03-11T21:49:31.252914+0000","last_unstale":"2024-03-11T21:50:09.306576+0000","last_undegraded":"2024-03-11T21:50:09.306576+0000","last_fullsized":"2024-03-11T21:50:09.306576+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:06:37.877498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:04.305599+0000","last_change":"2024-03-11T21:49:33.260054+0000","last_active":"2024-03-11T21:50:04.305599+0000","last_peered":"2024-03-11T21:50:04.305599+0000","last_clean":"2024-03-11T21:50:04.305599+0000","last_became_active":"2024-03-11T21:49:33.259454+0000","last_became_peered":"2024-03-11T21:49:33.259454+0000","last_unstale":"2024-03-11T21:50:04.305599+0000","last_undegraded":"2024-03-11T21:50:04.305599+0000","last_fullsized":"2024-03-11T21:50:04.305599+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:42:57.215099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702046+0000","last_change":"2024-03-11T21:49:35.266706+0000","last_active":"2024-03-11T21:49:58.702046+0000","last_peered":"2024-03-11T21:49:58.702046+0000","last_clean":"2024-03-11T21:49:58.702046+0000","last_became_active":"2024-03-11T21:49:35.265564+0000","last_became_peered":"2024-03-11T21:49:35.265564+0000","last_unstale":"2024-03-11T21:49:58.702046+0000","last_undegraded":"2024-03-11T21:49:58.702046+0000","last_fullsized":"2024-03-11T21:49:58.702046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:11:36.978228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702618+0000","last_change":"2024-03-11T21:49:29.250042+0000","last_active":"2024-03-11T21:49:58.702618+0000","last_peered":"2024-03-11T21:49:58.702618+0000","last_clean":"2024-03-11T21:49:58.702618+0000","last_became_active":"2024-03-11T21:49:29.249315+0000","last_became_peered":"2024-03-11T21:49:29.249315+0000","last_unstale":"2024-03-11T21:49:58.702618+0000","last_undegraded":"2024-03-11T21:49:58.702618+0000","last_fullsized":"2024-03-11T21:49:58.702618+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:14:46.543033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704296+0000","last_change":"2024-03-11T21:49:31.255012+0000","last_active":"2024-03-11T21:49:58.704296+0000","last_peered":"2024-03-11T21:49:58.704296+0000","last_clean":"2024-03-11T21:49:58.704296+0000","last_became_active":"2024-03-11T21:49:31.254549+0000","last_became_peered":"2024-03-11T21:49:31.254549+0000","last_unstale":"2024-03-11T21:49:58.704296+0000","last_undegraded":"2024-03-11T21:49:58.704296+0000","last_fullsized":"2024-03-11T21:49:58.704296+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:48:31.216797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:09.306500+0000","last_change":"2024-03-11T21:49:33.256818+0000","last_active":"2024-03-11T21:50:09.306500+0000","last_peered":"2024-03-11T21:50:09.306500+0000","last_clean":"2024-03-11T21:50:09.306500+0000","last_became_active":"2024-03-11T21:49:33.256358+0000","last_became_peered":"2024-03-11T21:49:33.256358+0000","last_unstale":"2024-03-11T21:50:09.306500+0000","last_undegraded":"2024-03-11T21:50:09.306500+0000","last_fullsized":"2024-03-11T21:50:09.306500+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:11:26.850640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702240+0000","last_change":"2024-03-11T21:49:35.265389+0000","last_active":"2024-03-11T21:49:58.702240+0000","last_peered":"2024-03-11T21:49:58.702240+0000","last_clean":"2024-03-11T21:49:58.702240+0000","last_became_active":"2024-03-11T21:49:35.265116+0000","last_became_peered":"2024-03-11T21:49:35.265116+0000","last_unstale":"2024-03-11T21:49:58.702240+0000","last_undegraded":"2024-03-11T21:49:58.702240+0000","last_fullsized":"2024-03-11T21:49:58.702240+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:31:58.732888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705236+0000","last_change":"2024-03-11T21:49:29.252811+0000","last_active":"2024-03-11T21:49:58.705236+0000","last_peered":"2024-03-11T21:49:58.705236+0000","last_clean":"2024-03-11T21:49:58.705236+0000","last_became_active":"2024-03-11T21:49:29.251957+0000","last_became_peered":"2024-03-11T21:49:29.251957+0000","last_unstale":"2024-03-11T21:49:58.705236+0000","last_undegraded":"2024-03-11T21:49:58.705236+0000","last_fullsized":"2024-03-11T21:49:58.705236+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:21:32.979820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703562+0000","last_change":"2024-03-11T21:49:31.254662+0000","last_active":"2024-03-11T21:49:58.703562+0000","last_peered":"2024-03-11T21:49:58.703562+0000","last_clean":"2024-03-11T21:49:58.703562+0000","last_became_active":"2024-03-11T21:49:31.254264+0000","last_became_peered":"2024-03-11T21:49:31.254264+0000","last_unstale":"2024-03-11T21:49:58.703562+0000","last_undegraded":"2024-03-11T21:49:58.703562+0000","last_fullsized":"2024-03-11T21:49:58.703562+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:41:05.509612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702919+0000","last_change":"2024-03-11T21:49:33.259801+0000","last_active":"2024-03-11T21:49:58.702919+0000","last_peered":"2024-03-11T21:49:58.702919+0000","last_clean":"2024-03-11T21:49:58.702919+0000","last_became_active":"2024-03-11T21:49:33.258815+0000","last_became_peered":"2024-03-11T21:49:33.258815+0000","last_unstale":"2024-03-11T21:49:58.702919+0000","last_undegraded":"2024-03-11T21:49:58.702919+0000","last_fullsized":"2024-03-11T21:49:58.702919+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:25:04.032931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704804+0000","last_change":"2024-03-11T21:49:35.267329+0000","last_active":"2024-03-11T21:49:58.704804+0000","last_peered":"2024-03-11T21:49:58.704804+0000","last_clean":"2024-03-11T21:49:58.704804+0000","last_became_active":"2024-03-11T21:49:35.266649+0000","last_became_peered":"2024-03-11T21:49:35.266649+0000","last_unstale":"2024-03-11T21:49:58.704804+0000","last_undegraded":"2024-03-11T21:49:58.704804+0000","last_fullsized":"2024-03-11T21:49:58.704804+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:58:29.358804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:04.305639+0000","last_change":"2024-03-11T21:49:33.259530+0000","last_active":"2024-03-11T21:50:04.305639+0000","last_peered":"2024-03-11T21:50:04.305639+0000","last_clean":"2024-03-11T21:50:04.305639+0000","last_became_active":"2024-03-11T21:49:33.258482+0000","last_became_peered":"2024-03-11T21:49:33.258482+0000","last_unstale":"2024-03-11T21:50:04.305639+0000","last_undegraded":"2024-03-11T21:50:04.305639+0000","last_fullsized":"2024-03-11T21:50:04.305639+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:59:25.773769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702251+0000","last_change":"2024-03-11T21:49:29.251650+0000","last_active":"2024-03-11T21:49:58.702251+0000","last_peered":"2024-03-11T21:49:58.702251+0000","last_clean":"2024-03-11T21:49:58.702251+0000","last_became_active":"2024-03-11T21:49:29.251347+0000","last_became_peered":"2024-03-11T21:49:29.251347+0000","last_unstale":"2024-03-11T21:49:58.702251+0000","last_undegraded":"2024-03-11T21:49:58.702251+0000","last_fullsized":"2024-03-11T21:49:58.702251+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:54.751525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703392+0000","last_change":"2024-03-11T21:49:31.251779+0000","last_active":"2024-03-11T21:49:58.703392+0000","last_peered":"2024-03-11T21:49:58.703392+0000","last_clean":"2024-03-11T21:49:58.703392+0000","last_became_active":"2024-03-11T21:49:31.251476+0000","last_became_peered":"2024-03-11T21:49:31.251476+0000","last_unstale":"2024-03-11T21:49:58.703392+0000","last_undegraded":"2024-03-11T21:49:58.703392+0000","last_fullsized":"2024-03-11T21:49:58.703392+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:49:29.319220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704064+0000","last_change":"2024-03-11T21:49:35.266420+0000","last_active":"2024-03-11T21:49:58.704064+0000","last_peered":"2024-03-11T21:49:58.704064+0000","last_clean":"2024-03-11T21:49:58.704064+0000","last_became_active":"2024-03-11T21:49:35.266091+0000","last_became_peered":"2024-03-11T21:49:35.266091+0000","last_unstale":"2024-03-11T21:49:58.704064+0000","last_undegraded":"2024-03-11T21:49:58.704064+0000","last_fullsized":"2024-03-11T21:49:58.704064+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:50:59.755863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703638+0000","last_change":"2024-03-11T21:49:33.263362+0000","last_active":"2024-03-11T21:49:58.703638+0000","last_peered":"2024-03-11T21:49:58.703638+0000","last_clean":"2024-03-11T21:49:58.703638+0000","last_became_active":"2024-03-11T21:49:33.263106+0000","last_became_peered":"2024-03-11T21:49:33.263106+0000","last_unstale":"2024-03-11T21:49:58.703638+0000","last_undegraded":"2024-03-11T21:49:58.703638+0000","last_fullsized":"2024-03-11T21:49:58.703638+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:10:44.439402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700959+0000","last_change":"2024-03-11T21:49:29.250338+0000","last_active":"2024-03-11T21:49:58.700959+0000","last_peered":"2024-03-11T21:49:58.700959+0000","last_clean":"2024-03-11T21:49:58.700959+0000","last_became_active":"2024-03-11T21:49:29.249817+0000","last_became_peered":"2024-03-11T21:49:29.249817+0000","last_unstale":"2024-03-11T21:49:58.700959+0000","last_undegraded":"2024-03-11T21:49:58.700959+0000","last_fullsized":"2024-03-11T21:49:58.700959+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:17:19.098682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702127+0000","last_change":"2024-03-11T21:49:31.252087+0000","last_active":"2024-03-11T21:49:58.702127+0000","last_peered":"2024-03-11T21:49:58.702127+0000","last_clean":"2024-03-11T21:49:58.702127+0000","last_became_active":"2024-03-11T21:49:31.251298+0000","last_became_peered":"2024-03-11T21:49:31.251298+0000","last_unstale":"2024-03-11T21:49:58.702127+0000","last_undegraded":"2024-03-11T21:49:58.702127+0000","last_fullsized":"2024-03-11T21:49:58.702127+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:32:12.796906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703679+0000","last_change":"2024-03-11T21:49:35.266294+0000","last_active":"2024-03-11T21:49:58.703679+0000","last_peered":"2024-03-11T21:49:58.703679+0000","last_clean":"2024-03-11T21:49:58.703679+0000","last_became_active":"2024-03-11T21:49:35.265988+0000","last_became_peered":"2024-03-11T21:49:35.265988+0000","last_unstale":"2024-03-11T21:49:58.703679+0000","last_undegraded":"2024-03-11T21:49:58.703679+0000","last_fullsized":"2024-03-11T21:49:58.703679+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:40:51.590266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703611+0000","last_change":"2024-03-11T21:49:29.252571+0000","last_active":"2024-03-11T21:49:58.703611+0000","last_peered":"2024-03-11T21:49:58.703611+0000","last_clean":"2024-03-11T21:49:58.703611+0000","last_became_active":"2024-03-11T21:49:29.251938+0000","last_became_peered":"2024-03-11T21:49:29.251938+0000","last_unstale":"2024-03-11T21:49:58.703611+0000","last_undegraded":"2024-03-11T21:49:58.703611+0000","last_fullsized":"2024-03-11T21:49:58.703611+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:24:18.723480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701554+0000","last_change":"2024-03-11T21:49:31.252435+0000","last_active":"2024-03-11T21:49:58.701554+0000","last_peered":"2024-03-11T21:49:58.701554+0000","last_clean":"2024-03-11T21:49:58.701554+0000","last_became_active":"2024-03-11T21:49:31.252055+0000","last_became_peered":"2024-03-11T21:49:31.252055+0000","last_unstale":"2024-03-11T21:49:58.701554+0000","last_undegraded":"2024-03-11T21:49:58.701554+0000","last_fullsized":"2024-03-11T21:49:58.701554+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:33:56.951573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703243+0000","last_change":"2024-03-11T21:49:33.259952+0000","last_active":"2024-03-11T21:49:58.703243+0000","last_peered":"2024-03-11T21:49:58.703243+0000","last_clean":"2024-03-11T21:49:58.703243+0000","last_became_active":"2024-03-11T21:49:33.259268+0000","last_became_peered":"2024-03-11T21:49:33.259268+0000","last_unstale":"2024-03-11T21:49:58.703243+0000","last_undegraded":"2024-03-11T21:49:58.703243+0000","last_fullsized":"2024-03-11T21:49:58.703243+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:26:18.012745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701948+0000","last_change":"2024-03-11T21:49:35.268219+0000","last_active":"2024-03-11T21:49:58.701948+0000","last_peered":"2024-03-11T21:49:58.701948+0000","last_clean":"2024-03-11T21:49:58.701948+0000","last_became_active":"2024-03-11T21:49:35.267860+0000","last_became_peered":"2024-03-11T21:49:35.267860+0000","last_unstale":"2024-03-11T21:49:58.701948+0000","last_undegraded":"2024-03-11T21:49:58.701948+0000","last_fullsized":"2024-03-11T21:49:58.701948+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:30:03.300547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:04.305607+0000","last_change":"2024-03-11T21:49:33.260136+0000","last_active":"2024-03-11T21:50:04.305607+0000","last_peered":"2024-03-11T21:50:04.305607+0000","last_clean":"2024-03-11T21:50:04.305607+0000","last_became_active":"2024-03-11T21:49:33.259528+0000","last_became_peered":"2024-03-11T21:49:33.259528+0000","last_unstale":"2024-03-11T21:50:04.305607+0000","last_undegraded":"2024-03-11T21:50:04.305607+0000","last_fullsized":"2024-03-11T21:50:04.305607+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:42:02.181473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704656+0000","last_change":"2024-03-11T21:49:29.254391+0000","last_active":"2024-03-11T21:49:58.704656+0000","last_peered":"2024-03-11T21:49:58.704656+0000","last_clean":"2024-03-11T21:49:58.704656+0000","last_became_active":"2024-03-11T21:49:29.252095+0000","last_became_peered":"2024-03-11T21:49:29.252095+0000","last_unstale":"2024-03-11T21:49:58.704656+0000","last_undegraded":"2024-03-11T21:49:58.704656+0000","last_fullsized":"2024-03-11T21:49:58.704656+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:29:27.311018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'19","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703151+0000","last_change":"2024-03-11T21:49:31.256561+0000","last_active":"2024-03-11T21:49:58.703151+0000","last_peered":"2024-03-11T21:49:58.703151+0000","last_clean":"2024-03-11T21:49:58.703151+0000","last_became_active":"2024-03-11T21:49:31.256225+0000","last_became_peered":"2024-03-11T21:49:31.256225+0000","last_unstale":"2024-03-11T21:49:58.703151+0000","last_undegraded":"2024-03-11T21:49:58.703151+0000","last_fullsized":"2024-03-11T21:49:58.703151+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:14:17.446682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703528+0000","last_change":"2024-03-11T21:49:35.269375+0000","last_active":"2024-03-11T21:49:58.703528+0000","last_peered":"2024-03-11T21:49:58.703528+0000","last_clean":"2024-03-11T21:49:58.703528+0000","last_became_active":"2024-03-11T21:49:35.269052+0000","last_became_peered":"2024-03-11T21:49:35.269052+0000","last_unstale":"2024-03-11T21:49:58.703528+0000","last_undegraded":"2024-03-11T21:49:58.703528+0000","last_fullsized":"2024-03-11T21:49:58.703528+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:04:02.650924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704366+0000","last_change":"2024-03-11T21:49:29.253121+0000","last_active":"2024-03-11T21:49:58.704366+0000","last_peered":"2024-03-11T21:49:58.704366+0000","last_clean":"2024-03-11T21:49:58.704366+0000","last_became_active":"2024-03-11T21:49:29.252761+0000","last_became_peered":"2024-03-11T21:49:29.252761+0000","last_unstale":"2024-03-11T21:49:58.704366+0000","last_undegraded":"2024-03-11T21:49:58.704366+0000","last_fullsized":"2024-03-11T21:49:58.704366+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:07:22.301323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703826+0000","last_change":"2024-03-11T21:49:31.254735+0000","last_active":"2024-03-11T21:49:58.703826+0000","last_peered":"2024-03-11T21:49:58.703826+0000","last_clean":"2024-03-11T21:49:58.703826+0000","last_became_active":"2024-03-11T21:49:31.253920+0000","last_became_peered":"2024-03-11T21:49:31.253920+0000","last_unstale":"2024-03-11T21:49:58.703826+0000","last_undegraded":"2024-03-11T21:49:58.703826+0000","last_fullsized":"2024-03-11T21:49:58.703826+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:46:16.897253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702522+0000","last_change":"2024-03-11T21:49:33.259568+0000","last_active":"2024-03-11T21:49:58.702522+0000","last_peered":"2024-03-11T21:49:58.702522+0000","last_clean":"2024-03-11T21:49:58.702522+0000","last_became_active":"2024-03-11T21:49:33.259214+0000","last_became_peered":"2024-03-11T21:49:33.259214+0000","last_unstale":"2024-03-11T21:49:58.702522+0000","last_undegraded":"2024-03-11T21:49:58.702522+0000","last_fullsized":"2024-03-11T21:49:58.702522+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:54:38.253162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700536+0000","last_change":"2024-03-11T21:49:35.264133+0000","last_active":"2024-03-11T21:49:58.700536+0000","last_peered":"2024-03-11T21:49:58.700536+0000","last_clean":"2024-03-11T21:49:58.700536+0000","last_became_active":"2024-03-11T21:49:35.263778+0000","last_became_peered":"2024-03-11T21:49:35.263778+0000","last_unstale":"2024-03-11T21:49:58.700536+0000","last_undegraded":"2024-03-11T21:49:58.700536+0000","last_fullsized":"2024-03-11T21:49:58.700536+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:53.404266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704974+0000","last_change":"2024-03-11T21:49:29.254789+0000","last_active":"2024-03-11T21:49:58.704974+0000","last_peered":"2024-03-11T21:49:58.704974+0000","last_clean":"2024-03-11T21:49:58.704974+0000","last_became_active":"2024-03-11T21:49:29.253502+0000","last_became_peered":"2024-03-11T21:49:29.253502+0000","last_unstale":"2024-03-11T21:49:58.704974+0000","last_undegraded":"2024-03-11T21:49:58.704974+0000","last_fullsized":"2024-03-11T21:49:58.704974+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:16:05.883121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703537+0000","last_change":"2024-03-11T21:49:31.255652+0000","last_active":"2024-03-11T21:49:58.703537+0000","last_peered":"2024-03-11T21:49:58.703537+0000","last_clean":"2024-03-11T21:49:58.703537+0000","last_became_active":"2024-03-11T21:49:31.254989+0000","last_became_peered":"2024-03-11T21:49:31.254989+0000","last_unstale":"2024-03-11T21:49:58.703537+0000","last_undegraded":"2024-03-11T21:49:58.703537+0000","last_fullsized":"2024-03-11T21:49:58.703537+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:34:56.202135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701041+0000","last_change":"2024-03-11T21:49:33.262443+0000","last_active":"2024-03-11T21:49:58.701041+0000","last_peered":"2024-03-11T21:49:58.701041+0000","last_clean":"2024-03-11T21:49:58.701041+0000","last_became_active":"2024-03-11T21:49:33.262045+0000","last_became_peered":"2024-03-11T21:49:33.262045+0000","last_unstale":"2024-03-11T21:49:58.701041+0000","last_undegraded":"2024-03-11T21:49:58.701041+0000","last_fullsized":"2024-03-11T21:49:58.701041+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:47:04.435813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702159+0000","last_change":"2024-03-11T21:49:35.263806+0000","last_active":"2024-03-11T21:49:58.702159+0000","last_peered":"2024-03-11T21:49:58.702159+0000","last_clean":"2024-03-11T21:49:58.702159+0000","last_became_active":"2024-03-11T21:49:35.263287+0000","last_became_peered":"2024-03-11T21:49:35.263287+0000","last_unstale":"2024-03-11T21:49:58.702159+0000","last_undegraded":"2024-03-11T21:49:58.702159+0000","last_fullsized":"2024-03-11T21:49:58.702159+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:30:47.721743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703431+0000","last_change":"2024-03-11T21:49:29.250345+0000","last_active":"2024-03-11T21:49:58.703431+0000","last_peered":"2024-03-11T21:49:58.703431+0000","last_clean":"2024-03-11T21:49:58.703431+0000","last_became_active":"2024-03-11T21:49:29.249600+0000","last_became_peered":"2024-03-11T21:49:29.249600+0000","last_unstale":"2024-03-11T21:49:58.703431+0000","last_undegraded":"2024-03-11T21:49:58.703431+0000","last_fullsized":"2024-03-11T21:49:58.703431+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:09:13.532627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705327+0000","last_change":"2024-03-11T21:49:31.255212+0000","last_active":"2024-03-11T21:49:58.705327+0000","last_peered":"2024-03-11T21:49:58.705327+0000","last_clean":"2024-03-11T21:49:58.705327+0000","last_became_active":"2024-03-11T21:49:31.254631+0000","last_became_peered":"2024-03-11T21:49:31.254631+0000","last_unstale":"2024-03-11T21:49:58.705327+0000","last_undegraded":"2024-03-11T21:49:58.705327+0000","last_fullsized":"2024-03-11T21:49:58.705327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:35:43.412087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702208+0000","last_change":"2024-03-11T21:49:33.259910+0000","last_active":"2024-03-11T21:49:58.702208+0000","last_peered":"2024-03-11T21:49:58.702208+0000","last_clean":"2024-03-11T21:49:58.702208+0000","last_became_active":"2024-03-11T21:49:33.259187+0000","last_became_peered":"2024-03-11T21:49:33.259187+0000","last_unstale":"2024-03-11T21:49:58.702208+0000","last_undegraded":"2024-03-11T21:49:58.702208+0000","last_fullsized":"2024-03-11T21:49:58.702208+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:19:32.487442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411342,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738392,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:50:16.996 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:50:17.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:17 smithi002 ceph-mon[25764]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:50:17.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:17 smithi122 ceph-mon[27585]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:50:17.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:17 smithi165 ceph-mon[27942]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:50:17.964 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:18.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:18 smithi002 ceph-mon[25764]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:18.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:18 smithi122 ceph-mon[27585]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:18.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:18 smithi165 ceph-mon[27942]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:20.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:20 smithi002 ceph-mon[25764]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:20.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:20 smithi122 ceph-mon[27585]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:20.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:20 smithi165 ceph-mon[27942]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:21.522 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:50:21.524 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-03-11T21:50:22.216 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-03-11T21:50:20.716708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96616,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126552,"statfs":{"total":287972524032,"available":287873589248,"internally_reserved":0,"allocated":3158016,"data_stored":2100345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002728"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702944+0000","last_change":"2024-03-11T21:49:35.266574+0000","last_active":"2024-03-11T21:49:58.702944+0000","last_peered":"2024-03-11T21:49:58.702944+0000","last_clean":"2024-03-11T21:49:58.702944+0000","last_became_active":"2024-03-11T21:49:35.264787+0000","last_became_peered":"2024-03-11T21:49:35.264787+0000","last_unstale":"2024-03-11T21:49:58.702944+0000","last_undegraded":"2024-03-11T21:49:58.702944+0000","last_fullsized":"2024-03-11T21:49:58.702944+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:11:33.021725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702826+0000","last_change":"2024-03-11T21:49:33.258448+0000","last_active":"2024-03-11T21:49:58.702826+0000","last_peered":"2024-03-11T21:49:58.702826+0000","last_clean":"2024-03-11T21:49:58.702826+0000","last_became_active":"2024-03-11T21:49:33.257803+0000","last_became_peered":"2024-03-11T21:49:33.257803+0000","last_unstale":"2024-03-11T21:49:58.702826+0000","last_undegraded":"2024-03-11T21:49:58.702826+0000","last_fullsized":"2024-03-11T21:49:58.702826+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:46:57.467404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703960+0000","last_change":"2024-03-11T21:49:29.250428+0000","last_active":"2024-03-11T21:49:58.703960+0000","last_peered":"2024-03-11T21:49:58.703960+0000","last_clean":"2024-03-11T21:49:58.703960+0000","last_became_active":"2024-03-11T21:49:29.250107+0000","last_became_peered":"2024-03-11T21:49:29.250107+0000","last_unstale":"2024-03-11T21:49:58.703960+0000","last_undegraded":"2024-03-11T21:49:58.703960+0000","last_fullsized":"2024-03-11T21:49:58.703960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:15:27.588120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702728+0000","last_change":"2024-03-11T21:49:31.254145+0000","last_active":"2024-03-11T21:49:58.702728+0000","last_peered":"2024-03-11T21:49:58.702728+0000","last_clean":"2024-03-11T21:49:58.702728+0000","last_became_active":"2024-03-11T21:49:31.253848+0000","last_became_peered":"2024-03-11T21:49:31.253848+0000","last_unstale":"2024-03-11T21:49:58.702728+0000","last_undegraded":"2024-03-11T21:49:58.702728+0000","last_fullsized":"2024-03-11T21:49:58.702728+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:20:40.354767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700876+0000","last_change":"2024-03-11T21:49:35.267766+0000","last_active":"2024-03-11T21:49:58.700876+0000","last_peered":"2024-03-11T21:49:58.700876+0000","last_clean":"2024-03-11T21:49:58.700876+0000","last_became_active":"2024-03-11T21:49:35.266910+0000","last_became_peered":"2024-03-11T21:49:35.266910+0000","last_unstale":"2024-03-11T21:49:58.700876+0000","last_undegraded":"2024-03-11T21:49:58.700876+0000","last_fullsized":"2024-03-11T21:49:58.700876+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:22:37.209896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702352+0000","last_change":"2024-03-11T21:49:33.259921+0000","last_active":"2024-03-11T21:49:58.702352+0000","last_peered":"2024-03-11T21:49:58.702352+0000","last_clean":"2024-03-11T21:49:58.702352+0000","last_became_active":"2024-03-11T21:49:33.257940+0000","last_became_peered":"2024-03-11T21:49:33.257940+0000","last_unstale":"2024-03-11T21:49:58.702352+0000","last_undegraded":"2024-03-11T21:49:58.702352+0000","last_fullsized":"2024-03-11T21:49:58.702352+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:40:43.861759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703785+0000","last_change":"2024-03-11T21:49:29.251137+0000","last_active":"2024-03-11T21:49:58.703785+0000","last_peered":"2024-03-11T21:49:58.703785+0000","last_clean":"2024-03-11T21:49:58.703785+0000","last_became_active":"2024-03-11T21:49:29.250423+0000","last_became_peered":"2024-03-11T21:49:29.250423+0000","last_unstale":"2024-03-11T21:49:58.703785+0000","last_undegraded":"2024-03-11T21:49:58.703785+0000","last_fullsized":"2024-03-11T21:49:58.703785+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:40:18.849565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703754+0000","last_change":"2024-03-11T21:49:31.254702+0000","last_active":"2024-03-11T21:49:58.703754+0000","last_peered":"2024-03-11T21:49:58.703754+0000","last_clean":"2024-03-11T21:49:58.703754+0000","last_became_active":"2024-03-11T21:49:31.254167+0000","last_became_peered":"2024-03-11T21:49:31.254167+0000","last_unstale":"2024-03-11T21:49:58.703754+0000","last_undegraded":"2024-03-11T21:49:58.703754+0000","last_fullsized":"2024-03-11T21:49:58.703754+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:12:36.172908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703010+0000","last_change":"2024-03-11T21:49:35.265377+0000","last_active":"2024-03-11T21:49:58.703010+0000","last_peered":"2024-03-11T21:49:58.703010+0000","last_clean":"2024-03-11T21:49:58.703010+0000","last_became_active":"2024-03-11T21:49:35.265078+0000","last_became_peered":"2024-03-11T21:49:35.265078+0000","last_unstale":"2024-03-11T21:49:58.703010+0000","last_undegraded":"2024-03-11T21:49:58.703010+0000","last_fullsized":"2024-03-11T21:49:58.703010+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:21:58.784825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703845+0000","last_change":"2024-03-11T21:49:33.258161+0000","last_active":"2024-03-11T21:49:58.703845+0000","last_peered":"2024-03-11T21:49:58.703845+0000","last_clean":"2024-03-11T21:49:58.703845+0000","last_became_active":"2024-03-11T21:49:33.257786+0000","last_became_peered":"2024-03-11T21:49:33.257786+0000","last_unstale":"2024-03-11T21:49:58.703845+0000","last_undegraded":"2024-03-11T21:49:58.703845+0000","last_fullsized":"2024-03-11T21:49:58.703845+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:17:14.095458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705414+0000","last_change":"2024-03-11T21:49:29.254414+0000","last_active":"2024-03-11T21:49:58.705414+0000","last_peered":"2024-03-11T21:49:58.705414+0000","last_clean":"2024-03-11T21:49:58.705414+0000","last_became_active":"2024-03-11T21:49:29.252615+0000","last_became_peered":"2024-03-11T21:49:29.252615+0000","last_unstale":"2024-03-11T21:49:58.705414+0000","last_undegraded":"2024-03-11T21:49:58.705414+0000","last_fullsized":"2024-03-11T21:49:58.705414+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:31:58.317393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704899+0000","last_change":"2024-03-11T21:49:31.255253+0000","last_active":"2024-03-11T21:49:58.704899+0000","last_peered":"2024-03-11T21:49:58.704899+0000","last_clean":"2024-03-11T21:49:58.704899+0000","last_became_active":"2024-03-11T21:49:31.254165+0000","last_became_peered":"2024-03-11T21:49:31.254165+0000","last_unstale":"2024-03-11T21:49:58.704899+0000","last_undegraded":"2024-03-11T21:49:58.704899+0000","last_fullsized":"2024-03-11T21:49:58.704899+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:58:19.179754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702575+0000","last_change":"2024-03-11T21:49:35.266190+0000","last_active":"2024-03-11T21:49:58.702575+0000","last_peered":"2024-03-11T21:49:58.702575+0000","last_clean":"2024-03-11T21:49:58.702575+0000","last_became_active":"2024-03-11T21:49:35.265820+0000","last_became_peered":"2024-03-11T21:49:35.265820+0000","last_unstale":"2024-03-11T21:49:58.702575+0000","last_undegraded":"2024-03-11T21:49:58.702575+0000","last_fullsized":"2024-03-11T21:49:58.702575+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:50.869654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702330+0000","last_change":"2024-03-11T21:49:33.258267+0000","last_active":"2024-03-11T21:49:58.702330+0000","last_peered":"2024-03-11T21:49:58.702330+0000","last_clean":"2024-03-11T21:49:58.702330+0000","last_became_active":"2024-03-11T21:49:33.257841+0000","last_became_peered":"2024-03-11T21:49:33.257841+0000","last_unstale":"2024-03-11T21:49:58.702330+0000","last_undegraded":"2024-03-11T21:49:58.702330+0000","last_fullsized":"2024-03-11T21:49:58.702330+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:27:22.694373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704092+0000","last_change":"2024-03-11T21:49:29.253450+0000","last_active":"2024-03-11T21:49:58.704092+0000","last_peered":"2024-03-11T21:49:58.704092+0000","last_clean":"2024-03-11T21:49:58.704092+0000","last_became_active":"2024-03-11T21:49:29.253065+0000","last_became_peered":"2024-03-11T21:49:29.253065+0000","last_unstale":"2024-03-11T21:49:58.704092+0000","last_undegraded":"2024-03-11T21:49:58.704092+0000","last_fullsized":"2024-03-11T21:49:58.704092+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:37:08.139190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705297+0000","last_change":"2024-03-11T21:49:31.252312+0000","last_active":"2024-03-11T21:49:58.705297+0000","last_peered":"2024-03-11T21:49:58.705297+0000","last_clean":"2024-03-11T21:49:58.705297+0000","last_became_active":"2024-03-11T21:49:31.251928+0000","last_became_peered":"2024-03-11T21:49:31.251928+0000","last_unstale":"2024-03-11T21:49:58.705297+0000","last_undegraded":"2024-03-11T21:49:58.705297+0000","last_fullsized":"2024-03-11T21:49:58.705297+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:02:01.747036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703137+0000","last_change":"2024-03-11T21:49:35.266507+0000","last_active":"2024-03-11T21:49:58.703137+0000","last_peered":"2024-03-11T21:49:58.703137+0000","last_clean":"2024-03-11T21:49:58.703137+0000","last_became_active":"2024-03-11T21:49:35.266074+0000","last_became_peered":"2024-03-11T21:49:35.266074+0000","last_unstale":"2024-03-11T21:49:58.703137+0000","last_undegraded":"2024-03-11T21:49:58.703137+0000","last_fullsized":"2024-03-11T21:49:58.703137+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:06:27.011361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704712+0000","last_change":"2024-03-11T21:49:33.259967+0000","last_active":"2024-03-11T21:49:58.704712+0000","last_peered":"2024-03-11T21:49:58.704712+0000","last_clean":"2024-03-11T21:49:58.704712+0000","last_became_active":"2024-03-11T21:49:33.259511+0000","last_became_peered":"2024-03-11T21:49:33.259511+0000","last_unstale":"2024-03-11T21:49:58.704712+0000","last_undegraded":"2024-03-11T21:49:58.704712+0000","last_fullsized":"2024-03-11T21:49:58.704712+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:17:54.755379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704789+0000","last_change":"2024-03-11T21:49:29.253158+0000","last_active":"2024-03-11T21:49:58.704789+0000","last_peered":"2024-03-11T21:49:58.704789+0000","last_clean":"2024-03-11T21:49:58.704789+0000","last_became_active":"2024-03-11T21:49:29.252094+0000","last_became_peered":"2024-03-11T21:49:29.252094+0000","last_unstale":"2024-03-11T21:49:58.704789+0000","last_undegraded":"2024-03-11T21:49:58.704789+0000","last_fullsized":"2024-03-11T21:49:58.704789+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:30:37.823365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703688+0000","last_change":"2024-03-11T21:49:31.251983+0000","last_active":"2024-03-11T21:49:58.703688+0000","last_peered":"2024-03-11T21:49:58.703688+0000","last_clean":"2024-03-11T21:49:58.703688+0000","last_became_active":"2024-03-11T21:49:31.251356+0000","last_became_peered":"2024-03-11T21:49:31.251356+0000","last_unstale":"2024-03-11T21:49:58.703688+0000","last_undegraded":"2024-03-11T21:49:58.703688+0000","last_fullsized":"2024-03-11T21:49:58.703688+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:36:24.329449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702025+0000","last_change":"2024-03-11T21:49:35.267124+0000","last_active":"2024-03-11T21:49:58.702025+0000","last_peered":"2024-03-11T21:49:58.702025+0000","last_clean":"2024-03-11T21:49:58.702025+0000","last_became_active":"2024-03-11T21:49:35.266210+0000","last_became_peered":"2024-03-11T21:49:35.266210+0000","last_unstale":"2024-03-11T21:49:58.702025+0000","last_undegraded":"2024-03-11T21:49:58.702025+0000","last_fullsized":"2024-03-11T21:49:58.702025+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:42:54.996928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704000+0000","last_change":"2024-03-11T21:49:33.259271+0000","last_active":"2024-03-11T21:49:58.704000+0000","last_peered":"2024-03-11T21:49:58.704000+0000","last_clean":"2024-03-11T21:49:58.704000+0000","last_became_active":"2024-03-11T21:49:33.258689+0000","last_became_peered":"2024-03-11T21:49:33.258689+0000","last_unstale":"2024-03-11T21:49:58.704000+0000","last_undegraded":"2024-03-11T21:49:58.704000+0000","last_fullsized":"2024-03-11T21:49:58.704000+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:38:48.583083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702733+0000","last_change":"2024-03-11T21:49:29.251212+0000","last_active":"2024-03-11T21:49:58.702733+0000","last_peered":"2024-03-11T21:49:58.702733+0000","last_clean":"2024-03-11T21:49:58.702733+0000","last_became_active":"2024-03-11T21:49:29.250181+0000","last_became_peered":"2024-03-11T21:49:29.250181+0000","last_unstale":"2024-03-11T21:49:58.702733+0000","last_undegraded":"2024-03-11T21:49:58.702733+0000","last_fullsized":"2024-03-11T21:49:58.702733+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:00:11.889227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'5","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704589+0000","last_change":"2024-03-11T21:49:31.251994+0000","last_active":"2024-03-11T21:49:58.704589+0000","last_peered":"2024-03-11T21:49:58.704589+0000","last_clean":"2024-03-11T21:49:58.704589+0000","last_became_active":"2024-03-11T21:49:31.251639+0000","last_became_peered":"2024-03-11T21:49:31.251639+0000","last_unstale":"2024-03-11T21:49:58.704589+0000","last_undegraded":"2024-03-11T21:49:58.704589+0000","last_fullsized":"2024-03-11T21:49:58.704589+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:48.857302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702082+0000","last_change":"2024-03-11T21:49:35.266794+0000","last_active":"2024-03-11T21:49:58.702082+0000","last_peered":"2024-03-11T21:49:58.702082+0000","last_clean":"2024-03-11T21:49:58.702082+0000","last_became_active":"2024-03-11T21:49:35.265366+0000","last_became_peered":"2024-03-11T21:49:35.265366+0000","last_unstale":"2024-03-11T21:49:58.702082+0000","last_undegraded":"2024-03-11T21:49:58.702082+0000","last_fullsized":"2024-03-11T21:49:58.702082+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:39:15.017312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:14.307724+0000","last_change":"2024-03-11T21:49:33.260199+0000","last_active":"2024-03-11T21:50:14.307724+0000","last_peered":"2024-03-11T21:50:14.307724+0000","last_clean":"2024-03-11T21:50:14.307724+0000","last_became_active":"2024-03-11T21:49:33.259086+0000","last_became_peered":"2024-03-11T21:49:33.259086+0000","last_unstale":"2024-03-11T21:50:14.307724+0000","last_undegraded":"2024-03-11T21:50:14.307724+0000","last_fullsized":"2024-03-11T21:50:14.307724+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:21:46.569479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701157+0000","last_change":"2024-03-11T21:49:29.250365+0000","last_active":"2024-03-11T21:49:58.701157+0000","last_peered":"2024-03-11T21:49:58.701157+0000","last_clean":"2024-03-11T21:49:58.701157+0000","last_became_active":"2024-03-11T21:49:29.250013+0000","last_became_peered":"2024-03-11T21:49:29.250013+0000","last_unstale":"2024-03-11T21:49:58.701157+0000","last_undegraded":"2024-03-11T21:49:58.701157+0000","last_fullsized":"2024-03-11T21:49:58.701157+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:42:10.961047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700746+0000","last_change":"2024-03-11T21:49:31.251657+0000","last_active":"2024-03-11T21:49:58.700746+0000","last_peered":"2024-03-11T21:49:58.700746+0000","last_clean":"2024-03-11T21:49:58.700746+0000","last_became_active":"2024-03-11T21:49:31.251270+0000","last_became_peered":"2024-03-11T21:49:31.251270+0000","last_unstale":"2024-03-11T21:49:58.700746+0000","last_undegraded":"2024-03-11T21:49:58.700746+0000","last_fullsized":"2024-03-11T21:49:58.700746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:50:42.839109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704600+0000","last_change":"2024-03-11T21:49:35.267448+0000","last_active":"2024-03-11T21:49:58.704600+0000","last_peered":"2024-03-11T21:49:58.704600+0000","last_clean":"2024-03-11T21:49:58.704600+0000","last_became_active":"2024-03-11T21:49:35.266795+0000","last_became_peered":"2024-03-11T21:49:35.266795+0000","last_unstale":"2024-03-11T21:49:58.704600+0000","last_undegraded":"2024-03-11T21:49:58.704600+0000","last_fullsized":"2024-03-11T21:49:58.704600+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:13:51.922059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702266+0000","last_change":"2024-03-11T21:49:33.262434+0000","last_active":"2024-03-11T21:49:58.702266+0000","last_peered":"2024-03-11T21:49:58.702266+0000","last_clean":"2024-03-11T21:49:58.702266+0000","last_became_active":"2024-03-11T21:49:33.261983+0000","last_became_peered":"2024-03-11T21:49:33.261983+0000","last_unstale":"2024-03-11T21:49:58.702266+0000","last_undegraded":"2024-03-11T21:49:58.702266+0000","last_fullsized":"2024-03-11T21:49:58.702266+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:31:08.122586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701995+0000","last_change":"2024-03-11T21:49:29.252638+0000","last_active":"2024-03-11T21:49:58.701995+0000","last_peered":"2024-03-11T21:49:58.701995+0000","last_clean":"2024-03-11T21:49:58.701995+0000","last_became_active":"2024-03-11T21:49:29.252053+0000","last_became_peered":"2024-03-11T21:49:29.252053+0000","last_unstale":"2024-03-11T21:49:58.701995+0000","last_undegraded":"2024-03-11T21:49:58.701995+0000","last_fullsized":"2024-03-11T21:49:58.701995+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:33:16.238062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700713+0000","last_change":"2024-03-11T21:49:31.251741+0000","last_active":"2024-03-11T21:49:58.700713+0000","last_peered":"2024-03-11T21:49:58.700713+0000","last_clean":"2024-03-11T21:49:58.700713+0000","last_became_active":"2024-03-11T21:49:31.251444+0000","last_became_peered":"2024-03-11T21:49:31.251444+0000","last_unstale":"2024-03-11T21:49:58.700713+0000","last_undegraded":"2024-03-11T21:49:58.700713+0000","last_fullsized":"2024-03-11T21:49:58.700713+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:13:10.243341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702655+0000","last_change":"2024-03-11T21:49:35.266395+0000","last_active":"2024-03-11T21:49:58.702655+0000","last_peered":"2024-03-11T21:49:58.702655+0000","last_clean":"2024-03-11T21:49:58.702655+0000","last_became_active":"2024-03-11T21:49:35.265703+0000","last_became_peered":"2024-03-11T21:49:35.265703+0000","last_unstale":"2024-03-11T21:49:58.702655+0000","last_undegraded":"2024-03-11T21:49:58.702655+0000","last_fullsized":"2024-03-11T21:49:58.702655+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:28:28.143541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700591+0000","last_change":"2024-03-11T21:49:33.256796+0000","last_active":"2024-03-11T21:49:58.700591+0000","last_peered":"2024-03-11T21:49:58.700591+0000","last_clean":"2024-03-11T21:49:58.700591+0000","last_became_active":"2024-03-11T21:49:33.256236+0000","last_became_peered":"2024-03-11T21:49:33.256236+0000","last_unstale":"2024-03-11T21:49:58.700591+0000","last_undegraded":"2024-03-11T21:49:58.700591+0000","last_fullsized":"2024-03-11T21:49:58.700591+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:14:24.408240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705549+0000","last_change":"2024-03-11T21:49:29.253555+0000","last_active":"2024-03-11T21:49:58.705549+0000","last_peered":"2024-03-11T21:49:58.705549+0000","last_clean":"2024-03-11T21:49:58.705549+0000","last_became_active":"2024-03-11T21:49:29.253166+0000","last_became_peered":"2024-03-11T21:49:29.253166+0000","last_unstale":"2024-03-11T21:49:58.705549+0000","last_undegraded":"2024-03-11T21:49:58.705549+0000","last_fullsized":"2024-03-11T21:49:58.705549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:50:15.090986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704956+0000","last_change":"2024-03-11T21:49:31.255719+0000","last_active":"2024-03-11T21:49:58.704956+0000","last_peered":"2024-03-11T21:49:58.704956+0000","last_clean":"2024-03-11T21:49:58.704956+0000","last_became_active":"2024-03-11T21:49:31.254802+0000","last_became_peered":"2024-03-11T21:49:31.254802+0000","last_unstale":"2024-03-11T21:49:58.704956+0000","last_undegraded":"2024-03-11T21:49:58.704956+0000","last_fullsized":"2024-03-11T21:49:58.704956+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:13:30.795360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702317+0000","last_change":"2024-03-11T21:49:35.264816+0000","last_active":"2024-03-11T21:49:58.702317+0000","last_peered":"2024-03-11T21:49:58.702317+0000","last_clean":"2024-03-11T21:49:58.702317+0000","last_became_active":"2024-03-11T21:49:35.263949+0000","last_became_peered":"2024-03-11T21:49:35.263949+0000","last_unstale":"2024-03-11T21:49:58.702317+0000","last_undegraded":"2024-03-11T21:49:58.702317+0000","last_fullsized":"2024-03-11T21:49:58.702317+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:15:15.392421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700948+0000","last_change":"2024-03-11T21:49:33.257120+0000","last_active":"2024-03-11T21:49:58.700948+0000","last_peered":"2024-03-11T21:49:58.700948+0000","last_clean":"2024-03-11T21:49:58.700948+0000","last_became_active":"2024-03-11T21:49:33.256827+0000","last_became_peered":"2024-03-11T21:49:33.256827+0000","last_unstale":"2024-03-11T21:49:58.700948+0000","last_undegraded":"2024-03-11T21:49:58.700948+0000","last_fullsized":"2024-03-11T21:49:58.700948+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:53:15.292904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704639+0000","last_change":"2024-03-11T21:49:29.253879+0000","last_active":"2024-03-11T21:49:58.704639+0000","last_peered":"2024-03-11T21:49:58.704639+0000","last_clean":"2024-03-11T21:49:58.704639+0000","last_became_active":"2024-03-11T21:49:29.253502+0000","last_became_peered":"2024-03-11T21:49:29.253502+0000","last_unstale":"2024-03-11T21:49:58.704639+0000","last_undegraded":"2024-03-11T21:49:58.704639+0000","last_fullsized":"2024-03-11T21:49:58.704639+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:54:11.900918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'6","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704176+0000","last_change":"2024-03-11T21:49:31.254462+0000","last_active":"2024-03-11T21:49:58.704176+0000","last_peered":"2024-03-11T21:49:58.704176+0000","last_clean":"2024-03-11T21:49:58.704176+0000","last_became_active":"2024-03-11T21:49:31.254114+0000","last_became_peered":"2024-03-11T21:49:31.254114+0000","last_unstale":"2024-03-11T21:49:58.704176+0000","last_undegraded":"2024-03-11T21:49:58.704176+0000","last_fullsized":"2024-03-11T21:49:58.704176+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:05:18.897284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703166+0000","last_change":"2024-03-11T21:49:35.266310+0000","last_active":"2024-03-11T21:49:58.703166+0000","last_peered":"2024-03-11T21:49:58.703166+0000","last_clean":"2024-03-11T21:49:58.703166+0000","last_became_active":"2024-03-11T21:49:35.265833+0000","last_became_peered":"2024-03-11T21:49:35.265833+0000","last_unstale":"2024-03-11T21:49:58.703166+0000","last_undegraded":"2024-03-11T21:49:58.703166+0000","last_fullsized":"2024-03-11T21:49:58.703166+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:23:08.633779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703954+0000","last_change":"2024-03-11T21:49:33.260132+0000","last_active":"2024-03-11T21:49:58.703954+0000","last_peered":"2024-03-11T21:49:58.703954+0000","last_clean":"2024-03-11T21:49:58.703954+0000","last_became_active":"2024-03-11T21:49:33.259118+0000","last_became_peered":"2024-03-11T21:49:33.259118+0000","last_unstale":"2024-03-11T21:49:58.703954+0000","last_undegraded":"2024-03-11T21:49:58.703954+0000","last_fullsized":"2024-03-11T21:49:58.703954+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:07:02.863183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705276+0000","last_change":"2024-03-11T21:49:29.252975+0000","last_active":"2024-03-11T21:49:58.705276+0000","last_peered":"2024-03-11T21:49:58.705276+0000","last_clean":"2024-03-11T21:49:58.705276+0000","last_became_active":"2024-03-11T21:49:29.252236+0000","last_became_peered":"2024-03-11T21:49:29.252236+0000","last_unstale":"2024-03-11T21:49:58.705276+0000","last_undegraded":"2024-03-11T21:49:58.705276+0000","last_fullsized":"2024-03-11T21:49:58.705276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:11:32.206643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704543+0000","last_change":"2024-03-11T21:49:31.255351+0000","last_active":"2024-03-11T21:49:58.704543+0000","last_peered":"2024-03-11T21:49:58.704543+0000","last_clean":"2024-03-11T21:49:58.704543+0000","last_became_active":"2024-03-11T21:49:31.254904+0000","last_became_peered":"2024-03-11T21:49:31.254904+0000","last_unstale":"2024-03-11T21:49:58.704543+0000","last_undegraded":"2024-03-11T21:49:58.704543+0000","last_fullsized":"2024-03-11T21:49:58.704543+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:13:57.471498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703346+0000","last_change":"2024-03-11T21:49:35.266079+0000","last_active":"2024-03-11T21:49:58.703346+0000","last_peered":"2024-03-11T21:49:58.703346+0000","last_clean":"2024-03-11T21:49:58.703346+0000","last_became_active":"2024-03-11T21:49:35.265750+0000","last_became_peered":"2024-03-11T21:49:35.265750+0000","last_unstale":"2024-03-11T21:49:58.703346+0000","last_undegraded":"2024-03-11T21:49:58.703346+0000","last_fullsized":"2024-03-11T21:49:58.703346+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:08:02.225852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701269+0000","last_change":"2024-03-11T21:49:33.257157+0000","last_active":"2024-03-11T21:49:58.701269+0000","last_peered":"2024-03-11T21:49:58.701269+0000","last_clean":"2024-03-11T21:49:58.701269+0000","last_became_active":"2024-03-11T21:49:33.256678+0000","last_became_peered":"2024-03-11T21:49:33.256678+0000","last_unstale":"2024-03-11T21:49:58.701269+0000","last_undegraded":"2024-03-11T21:49:58.701269+0000","last_fullsized":"2024-03-11T21:49:58.701269+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:13:41.924302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703449+0000","last_change":"2024-03-11T21:49:29.254065+0000","last_active":"2024-03-11T21:49:58.703449+0000","last_peered":"2024-03-11T21:49:58.703449+0000","last_clean":"2024-03-11T21:49:58.703449+0000","last_became_active":"2024-03-11T21:49:29.253496+0000","last_became_peered":"2024-03-11T21:49:29.253496+0000","last_unstale":"2024-03-11T21:49:58.703449+0000","last_undegraded":"2024-03-11T21:49:58.703449+0000","last_fullsized":"2024-03-11T21:49:58.703449+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:58:47.368433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705009+0000","last_change":"2024-03-11T21:49:31.251775+0000","last_active":"2024-03-11T21:49:58.705009+0000","last_peered":"2024-03-11T21:49:58.705009+0000","last_clean":"2024-03-11T21:49:58.705009+0000","last_became_active":"2024-03-11T21:49:31.251467+0000","last_became_peered":"2024-03-11T21:49:31.251467+0000","last_unstale":"2024-03-11T21:49:58.705009+0000","last_undegraded":"2024-03-11T21:49:58.705009+0000","last_fullsized":"2024-03-11T21:49:58.705009+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:53:17.227075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702268+0000","last_change":"2024-03-11T21:49:35.266496+0000","last_active":"2024-03-11T21:49:58.702268+0000","last_peered":"2024-03-11T21:49:58.702268+0000","last_clean":"2024-03-11T21:49:58.702268+0000","last_became_active":"2024-03-11T21:49:35.266167+0000","last_became_peered":"2024-03-11T21:49:35.266167+0000","last_unstale":"2024-03-11T21:49:58.702268+0000","last_undegraded":"2024-03-11T21:49:58.702268+0000","last_fullsized":"2024-03-11T21:49:58.702268+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:18:14.458344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:14.307703+0000","last_change":"2024-03-11T21:49:33.258873+0000","last_active":"2024-03-11T21:50:14.307703+0000","last_peered":"2024-03-11T21:50:14.307703+0000","last_clean":"2024-03-11T21:50:14.307703+0000","last_became_active":"2024-03-11T21:49:33.258554+0000","last_became_peered":"2024-03-11T21:49:33.258554+0000","last_unstale":"2024-03-11T21:50:14.307703+0000","last_undegraded":"2024-03-11T21:50:14.307703+0000","last_fullsized":"2024-03-11T21:50:14.307703+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:30:23.282288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704883+0000","last_change":"2024-03-11T21:49:29.254717+0000","last_active":"2024-03-11T21:49:58.704883+0000","last_peered":"2024-03-11T21:49:58.704883+0000","last_clean":"2024-03-11T21:49:58.704883+0000","last_became_active":"2024-03-11T21:49:29.253408+0000","last_became_peered":"2024-03-11T21:49:29.253408+0000","last_unstale":"2024-03-11T21:49:58.704883+0000","last_undegraded":"2024-03-11T21:49:58.704883+0000","last_fullsized":"2024-03-11T21:49:58.704883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:47:36.506975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701496+0000","last_change":"2024-03-11T21:49:31.252136+0000","last_active":"2024-03-11T21:49:58.701496+0000","last_peered":"2024-03-11T21:49:58.701496+0000","last_clean":"2024-03-11T21:49:58.701496+0000","last_became_active":"2024-03-11T21:49:31.251829+0000","last_became_peered":"2024-03-11T21:49:31.251829+0000","last_unstale":"2024-03-11T21:49:58.701496+0000","last_undegraded":"2024-03-11T21:49:58.701496+0000","last_fullsized":"2024-03-11T21:49:58.701496+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:11:35.422733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702953+0000","last_change":"2024-03-11T21:49:35.264377+0000","last_active":"2024-03-11T21:49:58.702953+0000","last_peered":"2024-03-11T21:49:58.702953+0000","last_clean":"2024-03-11T21:49:58.702953+0000","last_became_active":"2024-03-11T21:49:35.263974+0000","last_became_peered":"2024-03-11T21:49:35.263974+0000","last_unstale":"2024-03-11T21:49:58.702953+0000","last_undegraded":"2024-03-11T21:49:58.702953+0000","last_fullsized":"2024-03-11T21:49:58.702953+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:18:18.828987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:14.307754+0000","last_change":"2024-03-11T21:49:33.258616+0000","last_active":"2024-03-11T21:50:14.307754+0000","last_peered":"2024-03-11T21:50:14.307754+0000","last_clean":"2024-03-11T21:50:14.307754+0000","last_became_active":"2024-03-11T21:49:33.258280+0000","last_became_peered":"2024-03-11T21:49:33.258280+0000","last_unstale":"2024-03-11T21:50:14.307754+0000","last_undegraded":"2024-03-11T21:50:14.307754+0000","last_fullsized":"2024-03-11T21:50:14.307754+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:15:08.416440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703363+0000","last_change":"2024-03-11T21:49:29.254381+0000","last_active":"2024-03-11T21:49:58.703363+0000","last_peered":"2024-03-11T21:49:58.703363+0000","last_clean":"2024-03-11T21:49:58.703363+0000","last_became_active":"2024-03-11T21:49:29.253076+0000","last_became_peered":"2024-03-11T21:49:29.253076+0000","last_unstale":"2024-03-11T21:49:58.703363+0000","last_undegraded":"2024-03-11T21:49:58.703363+0000","last_fullsized":"2024-03-11T21:49:58.703363+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:40:17.066044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704810+0000","last_change":"2024-03-11T21:49:31.254237+0000","last_active":"2024-03-11T21:49:58.704810+0000","last_peered":"2024-03-11T21:49:58.704810+0000","last_clean":"2024-03-11T21:49:58.704810+0000","last_became_active":"2024-03-11T21:49:31.253830+0000","last_became_peered":"2024-03-11T21:49:31.253830+0000","last_unstale":"2024-03-11T21:49:58.704810+0000","last_undegraded":"2024-03-11T21:49:58.704810+0000","last_fullsized":"2024-03-11T21:49:58.704810+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:33:30.565887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703889+0000","last_change":"2024-03-11T21:49:35.264549+0000","last_active":"2024-03-11T21:49:58.703889+0000","last_peered":"2024-03-11T21:49:58.703889+0000","last_clean":"2024-03-11T21:49:58.703889+0000","last_became_active":"2024-03-11T21:49:35.264106+0000","last_became_peered":"2024-03-11T21:49:35.264106+0000","last_unstale":"2024-03-11T21:49:58.703889+0000","last_undegraded":"2024-03-11T21:49:58.703889+0000","last_fullsized":"2024-03-11T21:49:58.703889+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:40:58.738439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702669+0000","last_change":"2024-03-11T21:49:33.263001+0000","last_active":"2024-03-11T21:49:58.702669+0000","last_peered":"2024-03-11T21:49:58.702669+0000","last_clean":"2024-03-11T21:49:58.702669+0000","last_became_active":"2024-03-11T21:49:33.261811+0000","last_became_peered":"2024-03-11T21:49:33.261811+0000","last_unstale":"2024-03-11T21:49:58.702669+0000","last_undegraded":"2024-03-11T21:49:58.702669+0000","last_fullsized":"2024-03-11T21:49:58.702669+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:58.974435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702455+0000","last_change":"2024-03-11T21:49:29.253790+0000","last_active":"2024-03-11T21:49:58.702455+0000","last_peered":"2024-03-11T21:49:58.702455+0000","last_clean":"2024-03-11T21:49:58.702455+0000","last_became_active":"2024-03-11T21:49:29.253413+0000","last_became_peered":"2024-03-11T21:49:29.253413+0000","last_unstale":"2024-03-11T21:49:58.702455+0000","last_undegraded":"2024-03-11T21:49:58.702455+0000","last_fullsized":"2024-03-11T21:49:58.702455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:59:28.328961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702039+0000","last_change":"2024-03-11T21:49:31.254536+0000","last_active":"2024-03-11T21:49:58.702039+0000","last_peered":"2024-03-11T21:49:58.702039+0000","last_clean":"2024-03-11T21:49:58.702039+0000","last_became_active":"2024-03-11T21:49:31.253975+0000","last_became_peered":"2024-03-11T21:49:31.253975+0000","last_unstale":"2024-03-11T21:49:58.702039+0000","last_undegraded":"2024-03-11T21:49:58.702039+0000","last_fullsized":"2024-03-11T21:49:58.702039+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:50:23.411882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702390+0000","last_change":"2024-03-11T21:49:35.266623+0000","last_active":"2024-03-11T21:49:58.702390+0000","last_peered":"2024-03-11T21:49:58.702390+0000","last_clean":"2024-03-11T21:49:58.702390+0000","last_became_active":"2024-03-11T21:49:35.266263+0000","last_became_peered":"2024-03-11T21:49:35.266263+0000","last_unstale":"2024-03-11T21:49:58.702390+0000","last_undegraded":"2024-03-11T21:49:58.702390+0000","last_fullsized":"2024-03-11T21:49:58.702390+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:57:21.861934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702381+0000","last_change":"2024-03-11T21:49:33.262833+0000","last_active":"2024-03-11T21:49:58.702381+0000","last_peered":"2024-03-11T21:49:58.702381+0000","last_clean":"2024-03-11T21:49:58.702381+0000","last_became_active":"2024-03-11T21:49:33.262539+0000","last_became_peered":"2024-03-11T21:49:33.262539+0000","last_unstale":"2024-03-11T21:49:58.702381+0000","last_undegraded":"2024-03-11T21:49:58.702381+0000","last_fullsized":"2024-03-11T21:49:58.702381+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:51:04.157416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702066+0000","last_change":"2024-03-11T21:49:29.254501+0000","last_active":"2024-03-11T21:49:58.702066+0000","last_peered":"2024-03-11T21:49:58.702066+0000","last_clean":"2024-03-11T21:49:58.702066+0000","last_became_active":"2024-03-11T21:49:29.253334+0000","last_became_peered":"2024-03-11T21:49:29.253334+0000","last_unstale":"2024-03-11T21:49:58.702066+0000","last_undegraded":"2024-03-11T21:49:58.702066+0000","last_fullsized":"2024-03-11T21:49:58.702066+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:09:41.126405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703258+0000","last_change":"2024-03-11T21:49:31.251783+0000","last_active":"2024-03-11T21:49:58.703258+0000","last_peered":"2024-03-11T21:49:58.703258+0000","last_clean":"2024-03-11T21:49:58.703258+0000","last_became_active":"2024-03-11T21:49:31.251316+0000","last_became_peered":"2024-03-11T21:49:31.251316+0000","last_unstale":"2024-03-11T21:49:58.703258+0000","last_undegraded":"2024-03-11T21:49:58.703258+0000","last_fullsized":"2024-03-11T21:49:58.703258+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:03:10.579484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702308+0000","last_change":"2024-03-11T21:49:35.266224+0000","last_active":"2024-03-11T21:49:58.702308+0000","last_peered":"2024-03-11T21:49:58.702308+0000","last_clean":"2024-03-11T21:49:58.702308+0000","last_became_active":"2024-03-11T21:49:35.265890+0000","last_became_peered":"2024-03-11T21:49:35.265890+0000","last_unstale":"2024-03-11T21:49:58.702308+0000","last_undegraded":"2024-03-11T21:49:58.702308+0000","last_fullsized":"2024-03-11T21:49:58.702308+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:16:49.683222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704366+0000","last_change":"2024-03-11T21:49:29.252605+0000","last_active":"2024-03-11T21:49:58.704366+0000","last_peered":"2024-03-11T21:49:58.704366+0000","last_clean":"2024-03-11T21:49:58.704366+0000","last_became_active":"2024-03-11T21:49:29.252054+0000","last_became_peered":"2024-03-11T21:49:29.252054+0000","last_unstale":"2024-03-11T21:49:58.704366+0000","last_undegraded":"2024-03-11T21:49:58.704366+0000","last_fullsized":"2024-03-11T21:49:58.704366+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:04:42.686346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704492+0000","last_change":"2024-03-11T21:49:31.253985+0000","last_active":"2024-03-11T21:49:58.704492+0000","last_peered":"2024-03-11T21:49:58.704492+0000","last_clean":"2024-03-11T21:49:58.704492+0000","last_became_active":"2024-03-11T21:49:31.253665+0000","last_became_peered":"2024-03-11T21:49:31.253665+0000","last_unstale":"2024-03-11T21:49:58.704492+0000","last_undegraded":"2024-03-11T21:49:58.704492+0000","last_fullsized":"2024-03-11T21:49:58.704492+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:20:12.330839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:14.307793+0000","last_change":"2024-03-11T21:49:33.258937+0000","last_active":"2024-03-11T21:50:14.307793+0000","last_peered":"2024-03-11T21:50:14.307793+0000","last_clean":"2024-03-11T21:50:14.307793+0000","last_became_active":"2024-03-11T21:49:33.257996+0000","last_became_peered":"2024-03-11T21:49:33.257996+0000","last_unstale":"2024-03-11T21:50:14.307793+0000","last_undegraded":"2024-03-11T21:50:14.307793+0000","last_fullsized":"2024-03-11T21:50:14.307793+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:34:51.743830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700858+0000","last_change":"2024-03-11T21:49:35.265004+0000","last_active":"2024-03-11T21:49:58.700858+0000","last_peered":"2024-03-11T21:49:58.700858+0000","last_clean":"2024-03-11T21:49:58.700858+0000","last_became_active":"2024-03-11T21:49:35.264509+0000","last_became_peered":"2024-03-11T21:49:35.264509+0000","last_unstale":"2024-03-11T21:49:58.700858+0000","last_undegraded":"2024-03-11T21:49:58.700858+0000","last_fullsized":"2024-03-11T21:49:58.700858+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:46:13.626612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702551+0000","last_change":"2024-03-11T21:49:29.250000+0000","last_active":"2024-03-11T21:49:58.702551+0000","last_peered":"2024-03-11T21:49:58.702551+0000","last_clean":"2024-03-11T21:49:58.702551+0000","last_became_active":"2024-03-11T21:49:29.249114+0000","last_became_peered":"2024-03-11T21:49:29.249114+0000","last_unstale":"2024-03-11T21:49:58.702551+0000","last_undegraded":"2024-03-11T21:49:58.702551+0000","last_fullsized":"2024-03-11T21:49:58.702551+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:16:31.002441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:14.307747+0000","last_change":"2024-03-11T21:49:31.254374+0000","last_active":"2024-03-11T21:50:14.307747+0000","last_peered":"2024-03-11T21:50:14.307747+0000","last_clean":"2024-03-11T21:50:14.307747+0000","last_became_active":"2024-03-11T21:49:31.253872+0000","last_became_peered":"2024-03-11T21:49:31.253872+0000","last_unstale":"2024-03-11T21:50:14.307747+0000","last_undegraded":"2024-03-11T21:50:14.307747+0000","last_fullsized":"2024-03-11T21:50:14.307747+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:57:49.568838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703206+0000","last_change":"2024-03-11T21:49:33.259579+0000","last_active":"2024-03-11T21:49:58.703206+0000","last_peered":"2024-03-11T21:49:58.703206+0000","last_clean":"2024-03-11T21:49:58.703206+0000","last_became_active":"2024-03-11T21:49:33.258743+0000","last_became_peered":"2024-03-11T21:49:33.258743+0000","last_unstale":"2024-03-11T21:49:58.703206+0000","last_undegraded":"2024-03-11T21:49:58.703206+0000","last_fullsized":"2024-03-11T21:49:58.703206+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:43:13.877407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701610+0000","last_change":"2024-03-11T21:49:35.268647+0000","last_active":"2024-03-11T21:49:58.701610+0000","last_peered":"2024-03-11T21:49:58.701610+0000","last_clean":"2024-03-11T21:49:58.701610+0000","last_became_active":"2024-03-11T21:49:35.268328+0000","last_became_peered":"2024-03-11T21:49:35.268328+0000","last_unstale":"2024-03-11T21:49:58.701610+0000","last_undegraded":"2024-03-11T21:49:58.701610+0000","last_fullsized":"2024-03-11T21:49:58.701610+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:36:22.813279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703045+0000","last_change":"2024-03-11T21:49:29.251362+0000","last_active":"2024-03-11T21:49:58.703045+0000","last_peered":"2024-03-11T21:49:58.703045+0000","last_clean":"2024-03-11T21:49:58.703045+0000","last_became_active":"2024-03-11T21:49:29.250261+0000","last_became_peered":"2024-03-11T21:49:29.250261+0000","last_unstale":"2024-03-11T21:49:58.703045+0000","last_undegraded":"2024-03-11T21:49:58.703045+0000","last_fullsized":"2024-03-11T21:49:58.703045+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:12:59.338161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'19","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704963+0000","last_change":"2024-03-11T21:49:31.256262+0000","last_active":"2024-03-11T21:49:58.704963+0000","last_peered":"2024-03-11T21:49:58.704963+0000","last_clean":"2024-03-11T21:49:58.704963+0000","last_became_active":"2024-03-11T21:49:31.255764+0000","last_became_peered":"2024-03-11T21:49:31.255764+0000","last_unstale":"2024-03-11T21:49:58.704963+0000","last_undegraded":"2024-03-11T21:49:58.704963+0000","last_fullsized":"2024-03-11T21:49:58.704963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:17:36.847530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702318+0000","last_change":"2024-03-11T21:49:33.263685+0000","last_active":"2024-03-11T21:49:58.702318+0000","last_peered":"2024-03-11T21:49:58.702318+0000","last_clean":"2024-03-11T21:49:58.702318+0000","last_became_active":"2024-03-11T21:49:33.263404+0000","last_became_peered":"2024-03-11T21:49:33.263404+0000","last_unstale":"2024-03-11T21:49:58.702318+0000","last_undegraded":"2024-03-11T21:49:58.702318+0000","last_fullsized":"2024-03-11T21:49:58.702318+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:29:55.523250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703889+0000","last_change":"2024-03-11T21:49:35.268160+0000","last_active":"2024-03-11T21:49:58.703889+0000","last_peered":"2024-03-11T21:49:58.703889+0000","last_clean":"2024-03-11T21:49:58.703889+0000","last_became_active":"2024-03-11T21:49:35.267813+0000","last_became_peered":"2024-03-11T21:49:35.267813+0000","last_unstale":"2024-03-11T21:49:58.703889+0000","last_undegraded":"2024-03-11T21:49:58.703889+0000","last_fullsized":"2024-03-11T21:49:58.703889+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:19:59.956587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'18","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701871+0000","last_change":"2024-03-11T21:49:31.253876+0000","last_active":"2024-03-11T21:49:58.701871+0000","last_peered":"2024-03-11T21:49:58.701871+0000","last_clean":"2024-03-11T21:49:58.701871+0000","last_became_active":"2024-03-11T21:49:31.253542+0000","last_became_peered":"2024-03-11T21:49:31.253542+0000","last_unstale":"2024-03-11T21:49:58.701871+0000","last_undegraded":"2024-03-11T21:49:58.701871+0000","last_fullsized":"2024-03-11T21:49:58.701871+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:01:58.766660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700853+0000","last_change":"2024-03-11T21:49:29.250112+0000","last_active":"2024-03-11T21:49:58.700853+0000","last_peered":"2024-03-11T21:49:58.700853+0000","last_clean":"2024-03-11T21:49:58.700853+0000","last_became_active":"2024-03-11T21:49:29.249303+0000","last_became_peered":"2024-03-11T21:49:29.249303+0000","last_unstale":"2024-03-11T21:49:58.700853+0000","last_undegraded":"2024-03-11T21:49:58.700853+0000","last_fullsized":"2024-03-11T21:49:58.700853+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:39:04.972663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704108+0000","last_change":"2024-03-11T21:49:33.264210+0000","last_active":"2024-03-11T21:49:58.704108+0000","last_peered":"2024-03-11T21:49:58.704108+0000","last_clean":"2024-03-11T21:49:58.704108+0000","last_became_active":"2024-03-11T21:49:33.263904+0000","last_became_peered":"2024-03-11T21:49:33.263904+0000","last_unstale":"2024-03-11T21:49:58.704108+0000","last_undegraded":"2024-03-11T21:49:58.704108+0000","last_fullsized":"2024-03-11T21:49:58.704108+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:42.991166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703232+0000","last_change":"2024-03-11T21:49:35.269428+0000","last_active":"2024-03-11T21:49:58.703232+0000","last_peered":"2024-03-11T21:49:58.703232+0000","last_clean":"2024-03-11T21:49:58.703232+0000","last_became_active":"2024-03-11T21:49:35.269045+0000","last_became_peered":"2024-03-11T21:49:35.269045+0000","last_unstale":"2024-03-11T21:49:58.703232+0000","last_undegraded":"2024-03-11T21:49:58.703232+0000","last_fullsized":"2024-03-11T21:49:58.703232+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:21:55.158165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701686+0000","last_change":"2024-03-11T21:49:31.252365+0000","last_active":"2024-03-11T21:49:58.701686+0000","last_peered":"2024-03-11T21:49:58.701686+0000","last_clean":"2024-03-11T21:49:58.701686+0000","last_became_active":"2024-03-11T21:49:31.251505+0000","last_became_peered":"2024-03-11T21:49:31.251505+0000","last_unstale":"2024-03-11T21:49:58.701686+0000","last_undegraded":"2024-03-11T21:49:58.701686+0000","last_fullsized":"2024-03-11T21:49:58.701686+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:10:28.791536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704762+0000","last_change":"2024-03-11T21:49:29.252578+0000","last_active":"2024-03-11T21:49:58.704762+0000","last_peered":"2024-03-11T21:49:58.704762+0000","last_clean":"2024-03-11T21:49:58.704762+0000","last_became_active":"2024-03-11T21:49:29.251806+0000","last_became_peered":"2024-03-11T21:49:29.251806+0000","last_unstale":"2024-03-11T21:49:58.704762+0000","last_undegraded":"2024-03-11T21:49:58.704762+0000","last_fullsized":"2024-03-11T21:49:58.704762+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:53:15.456530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:00.807194+0000","last_change":"2024-03-11T21:49:17.908555+0000","last_active":"2024-03-11T21:50:00.807194+0000","last_peered":"2024-03-11T21:50:00.807194+0000","last_clean":"2024-03-11T21:50:00.807194+0000","last_became_active":"2024-03-11T21:49:17.908245+0000","last_became_peered":"2024-03-11T21:49:17.908245+0000","last_unstale":"2024-03-11T21:50:00.807194+0000","last_undegraded":"2024-03-11T21:50:00.807194+0000","last_fullsized":"2024-03-11T21:50:00.807194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:16.898621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:16.898621+0000","last_clean_scrub_stamp":"2024-03-11T21:49:16.898621+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:22:46.435509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703385+0000","last_change":"2024-03-11T21:49:33.259270+0000","last_active":"2024-03-11T21:49:58.703385+0000","last_peered":"2024-03-11T21:49:58.703385+0000","last_clean":"2024-03-11T21:49:58.703385+0000","last_became_active":"2024-03-11T21:49:33.258951+0000","last_became_peered":"2024-03-11T21:49:33.258951+0000","last_unstale":"2024-03-11T21:49:58.703385+0000","last_undegraded":"2024-03-11T21:49:58.703385+0000","last_fullsized":"2024-03-11T21:49:58.703385+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:34.972825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705392+0000","last_change":"2024-03-11T21:49:35.267216+0000","last_active":"2024-03-11T21:49:58.705392+0000","last_peered":"2024-03-11T21:49:58.705392+0000","last_clean":"2024-03-11T21:49:58.705392+0000","last_became_active":"2024-03-11T21:49:35.266225+0000","last_became_peered":"2024-03-11T21:49:35.266225+0000","last_unstale":"2024-03-11T21:49:58.705392+0000","last_undegraded":"2024-03-11T21:49:58.705392+0000","last_fullsized":"2024-03-11T21:49:58.705392+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:51:42.373614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'14","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701163+0000","last_change":"2024-03-11T21:49:31.251272+0000","last_active":"2024-03-11T21:49:58.701163+0000","last_peered":"2024-03-11T21:49:58.701163+0000","last_clean":"2024-03-11T21:49:58.701163+0000","last_became_active":"2024-03-11T21:49:31.250969+0000","last_became_peered":"2024-03-11T21:49:31.250969+0000","last_unstale":"2024-03-11T21:49:58.701163+0000","last_undegraded":"2024-03-11T21:49:58.701163+0000","last_fullsized":"2024-03-11T21:49:58.701163+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:53:14.801198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702137+0000","last_change":"2024-03-11T21:49:29.253080+0000","last_active":"2024-03-11T21:49:58.702137+0000","last_peered":"2024-03-11T21:49:58.702137+0000","last_clean":"2024-03-11T21:49:58.702137+0000","last_became_active":"2024-03-11T21:49:29.252371+0000","last_became_peered":"2024-03-11T21:49:29.252371+0000","last_unstale":"2024-03-11T21:49:58.702137+0000","last_undegraded":"2024-03-11T21:49:58.702137+0000","last_fullsized":"2024-03-11T21:49:58.702137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:25.249133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703764+0000","last_change":"2024-03-11T21:49:33.258715+0000","last_active":"2024-03-11T21:49:58.703764+0000","last_peered":"2024-03-11T21:49:58.703764+0000","last_clean":"2024-03-11T21:49:58.703764+0000","last_became_active":"2024-03-11T21:49:33.258402+0000","last_became_peered":"2024-03-11T21:49:33.258402+0000","last_unstale":"2024-03-11T21:49:58.703764+0000","last_undegraded":"2024-03-11T21:49:58.703764+0000","last_fullsized":"2024-03-11T21:49:58.703764+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:04:29.126653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702684+0000","last_change":"2024-03-11T21:49:35.266635+0000","last_active":"2024-03-11T21:49:58.702684+0000","last_peered":"2024-03-11T21:49:58.702684+0000","last_clean":"2024-03-11T21:49:58.702684+0000","last_became_active":"2024-03-11T21:49:35.265618+0000","last_became_peered":"2024-03-11T21:49:35.265618+0000","last_unstale":"2024-03-11T21:49:58.702684+0000","last_undegraded":"2024-03-11T21:49:58.702684+0000","last_fullsized":"2024-03-11T21:49:58.702684+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:12:21.022976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705195+0000","last_change":"2024-03-11T21:49:29.254818+0000","last_active":"2024-03-11T21:49:58.705195+0000","last_peered":"2024-03-11T21:49:58.705195+0000","last_clean":"2024-03-11T21:49:58.705195+0000","last_became_active":"2024-03-11T21:49:29.253609+0000","last_became_peered":"2024-03-11T21:49:29.253609+0000","last_unstale":"2024-03-11T21:49:58.705195+0000","last_undegraded":"2024-03-11T21:49:58.705195+0000","last_fullsized":"2024-03-11T21:49:58.705195+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:26:41.232304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'28","reported_seq":82,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:14.307540+0000","last_change":"2024-03-11T21:49:31.253200+0000","last_active":"2024-03-11T21:50:14.307540+0000","last_peered":"2024-03-11T21:50:14.307540+0000","last_clean":"2024-03-11T21:50:14.307540+0000","last_became_active":"2024-03-11T21:49:31.252914+0000","last_became_peered":"2024-03-11T21:49:31.252914+0000","last_unstale":"2024-03-11T21:50:14.307540+0000","last_undegraded":"2024-03-11T21:50:14.307540+0000","last_fullsized":"2024-03-11T21:50:14.307540+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:06:37.877498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:14.307687+0000","last_change":"2024-03-11T21:49:33.260054+0000","last_active":"2024-03-11T21:50:14.307687+0000","last_peered":"2024-03-11T21:50:14.307687+0000","last_clean":"2024-03-11T21:50:14.307687+0000","last_became_active":"2024-03-11T21:49:33.259454+0000","last_became_peered":"2024-03-11T21:49:33.259454+0000","last_unstale":"2024-03-11T21:50:14.307687+0000","last_undegraded":"2024-03-11T21:50:14.307687+0000","last_fullsized":"2024-03-11T21:50:14.307687+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:42:57.215099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702046+0000","last_change":"2024-03-11T21:49:35.266706+0000","last_active":"2024-03-11T21:49:58.702046+0000","last_peered":"2024-03-11T21:49:58.702046+0000","last_clean":"2024-03-11T21:49:58.702046+0000","last_became_active":"2024-03-11T21:49:35.265564+0000","last_became_peered":"2024-03-11T21:49:35.265564+0000","last_unstale":"2024-03-11T21:49:58.702046+0000","last_undegraded":"2024-03-11T21:49:58.702046+0000","last_fullsized":"2024-03-11T21:49:58.702046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:11:36.978228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702618+0000","last_change":"2024-03-11T21:49:29.250042+0000","last_active":"2024-03-11T21:49:58.702618+0000","last_peered":"2024-03-11T21:49:58.702618+0000","last_clean":"2024-03-11T21:49:58.702618+0000","last_became_active":"2024-03-11T21:49:29.249315+0000","last_became_peered":"2024-03-11T21:49:29.249315+0000","last_unstale":"2024-03-11T21:49:58.702618+0000","last_undegraded":"2024-03-11T21:49:58.702618+0000","last_fullsized":"2024-03-11T21:49:58.702618+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:14:46.543033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704296+0000","last_change":"2024-03-11T21:49:31.255012+0000","last_active":"2024-03-11T21:49:58.704296+0000","last_peered":"2024-03-11T21:49:58.704296+0000","last_clean":"2024-03-11T21:49:58.704296+0000","last_became_active":"2024-03-11T21:49:31.254549+0000","last_became_peered":"2024-03-11T21:49:31.254549+0000","last_unstale":"2024-03-11T21:49:58.704296+0000","last_undegraded":"2024-03-11T21:49:58.704296+0000","last_fullsized":"2024-03-11T21:49:58.704296+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:48:31.216797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:14.307537+0000","last_change":"2024-03-11T21:49:33.256818+0000","last_active":"2024-03-11T21:50:14.307537+0000","last_peered":"2024-03-11T21:50:14.307537+0000","last_clean":"2024-03-11T21:50:14.307537+0000","last_became_active":"2024-03-11T21:49:33.256358+0000","last_became_peered":"2024-03-11T21:49:33.256358+0000","last_unstale":"2024-03-11T21:50:14.307537+0000","last_undegraded":"2024-03-11T21:50:14.307537+0000","last_fullsized":"2024-03-11T21:50:14.307537+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:11:26.850640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702240+0000","last_change":"2024-03-11T21:49:35.265389+0000","last_active":"2024-03-11T21:49:58.702240+0000","last_peered":"2024-03-11T21:49:58.702240+0000","last_clean":"2024-03-11T21:49:58.702240+0000","last_became_active":"2024-03-11T21:49:35.265116+0000","last_became_peered":"2024-03-11T21:49:35.265116+0000","last_unstale":"2024-03-11T21:49:58.702240+0000","last_undegraded":"2024-03-11T21:49:58.702240+0000","last_fullsized":"2024-03-11T21:49:58.702240+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:31:58.732888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705236+0000","last_change":"2024-03-11T21:49:29.252811+0000","last_active":"2024-03-11T21:49:58.705236+0000","last_peered":"2024-03-11T21:49:58.705236+0000","last_clean":"2024-03-11T21:49:58.705236+0000","last_became_active":"2024-03-11T21:49:29.251957+0000","last_became_peered":"2024-03-11T21:49:29.251957+0000","last_unstale":"2024-03-11T21:49:58.705236+0000","last_undegraded":"2024-03-11T21:49:58.705236+0000","last_fullsized":"2024-03-11T21:49:58.705236+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:21:32.979820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703562+0000","last_change":"2024-03-11T21:49:31.254662+0000","last_active":"2024-03-11T21:49:58.703562+0000","last_peered":"2024-03-11T21:49:58.703562+0000","last_clean":"2024-03-11T21:49:58.703562+0000","last_became_active":"2024-03-11T21:49:31.254264+0000","last_became_peered":"2024-03-11T21:49:31.254264+0000","last_unstale":"2024-03-11T21:49:58.703562+0000","last_undegraded":"2024-03-11T21:49:58.703562+0000","last_fullsized":"2024-03-11T21:49:58.703562+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:41:05.509612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702919+0000","last_change":"2024-03-11T21:49:33.259801+0000","last_active":"2024-03-11T21:49:58.702919+0000","last_peered":"2024-03-11T21:49:58.702919+0000","last_clean":"2024-03-11T21:49:58.702919+0000","last_became_active":"2024-03-11T21:49:33.258815+0000","last_became_peered":"2024-03-11T21:49:33.258815+0000","last_unstale":"2024-03-11T21:49:58.702919+0000","last_undegraded":"2024-03-11T21:49:58.702919+0000","last_fullsized":"2024-03-11T21:49:58.702919+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:25:04.032931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704804+0000","last_change":"2024-03-11T21:49:35.267329+0000","last_active":"2024-03-11T21:49:58.704804+0000","last_peered":"2024-03-11T21:49:58.704804+0000","last_clean":"2024-03-11T21:49:58.704804+0000","last_became_active":"2024-03-11T21:49:35.266649+0000","last_became_peered":"2024-03-11T21:49:35.266649+0000","last_unstale":"2024-03-11T21:49:58.704804+0000","last_undegraded":"2024-03-11T21:49:58.704804+0000","last_fullsized":"2024-03-11T21:49:58.704804+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:58:29.358804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:14.307723+0000","last_change":"2024-03-11T21:49:33.259530+0000","last_active":"2024-03-11T21:50:14.307723+0000","last_peered":"2024-03-11T21:50:14.307723+0000","last_clean":"2024-03-11T21:50:14.307723+0000","last_became_active":"2024-03-11T21:49:33.258482+0000","last_became_peered":"2024-03-11T21:49:33.258482+0000","last_unstale":"2024-03-11T21:50:14.307723+0000","last_undegraded":"2024-03-11T21:50:14.307723+0000","last_fullsized":"2024-03-11T21:50:14.307723+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:59:25.773769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702251+0000","last_change":"2024-03-11T21:49:29.251650+0000","last_active":"2024-03-11T21:49:58.702251+0000","last_peered":"2024-03-11T21:49:58.702251+0000","last_clean":"2024-03-11T21:49:58.702251+0000","last_became_active":"2024-03-11T21:49:29.251347+0000","last_became_peered":"2024-03-11T21:49:29.251347+0000","last_unstale":"2024-03-11T21:49:58.702251+0000","last_undegraded":"2024-03-11T21:49:58.702251+0000","last_fullsized":"2024-03-11T21:49:58.702251+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:54.751525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703392+0000","last_change":"2024-03-11T21:49:31.251779+0000","last_active":"2024-03-11T21:49:58.703392+0000","last_peered":"2024-03-11T21:49:58.703392+0000","last_clean":"2024-03-11T21:49:58.703392+0000","last_became_active":"2024-03-11T21:49:31.251476+0000","last_became_peered":"2024-03-11T21:49:31.251476+0000","last_unstale":"2024-03-11T21:49:58.703392+0000","last_undegraded":"2024-03-11T21:49:58.703392+0000","last_fullsized":"2024-03-11T21:49:58.703392+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:49:29.319220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704064+0000","last_change":"2024-03-11T21:49:35.266420+0000","last_active":"2024-03-11T21:49:58.704064+0000","last_peered":"2024-03-11T21:49:58.704064+0000","last_clean":"2024-03-11T21:49:58.704064+0000","last_became_active":"2024-03-11T21:49:35.266091+0000","last_became_peered":"2024-03-11T21:49:35.266091+0000","last_unstale":"2024-03-11T21:49:58.704064+0000","last_undegraded":"2024-03-11T21:49:58.704064+0000","last_fullsized":"2024-03-11T21:49:58.704064+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:50:59.755863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703638+0000","last_change":"2024-03-11T21:49:33.263362+0000","last_active":"2024-03-11T21:49:58.703638+0000","last_peered":"2024-03-11T21:49:58.703638+0000","last_clean":"2024-03-11T21:49:58.703638+0000","last_became_active":"2024-03-11T21:49:33.263106+0000","last_became_peered":"2024-03-11T21:49:33.263106+0000","last_unstale":"2024-03-11T21:49:58.703638+0000","last_undegraded":"2024-03-11T21:49:58.703638+0000","last_fullsized":"2024-03-11T21:49:58.703638+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:10:44.439402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700959+0000","last_change":"2024-03-11T21:49:29.250338+0000","last_active":"2024-03-11T21:49:58.700959+0000","last_peered":"2024-03-11T21:49:58.700959+0000","last_clean":"2024-03-11T21:49:58.700959+0000","last_became_active":"2024-03-11T21:49:29.249817+0000","last_became_peered":"2024-03-11T21:49:29.249817+0000","last_unstale":"2024-03-11T21:49:58.700959+0000","last_undegraded":"2024-03-11T21:49:58.700959+0000","last_fullsized":"2024-03-11T21:49:58.700959+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:17:19.098682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702127+0000","last_change":"2024-03-11T21:49:31.252087+0000","last_active":"2024-03-11T21:49:58.702127+0000","last_peered":"2024-03-11T21:49:58.702127+0000","last_clean":"2024-03-11T21:49:58.702127+0000","last_became_active":"2024-03-11T21:49:31.251298+0000","last_became_peered":"2024-03-11T21:49:31.251298+0000","last_unstale":"2024-03-11T21:49:58.702127+0000","last_undegraded":"2024-03-11T21:49:58.702127+0000","last_fullsized":"2024-03-11T21:49:58.702127+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:32:12.796906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703679+0000","last_change":"2024-03-11T21:49:35.266294+0000","last_active":"2024-03-11T21:49:58.703679+0000","last_peered":"2024-03-11T21:49:58.703679+0000","last_clean":"2024-03-11T21:49:58.703679+0000","last_became_active":"2024-03-11T21:49:35.265988+0000","last_became_peered":"2024-03-11T21:49:35.265988+0000","last_unstale":"2024-03-11T21:49:58.703679+0000","last_undegraded":"2024-03-11T21:49:58.703679+0000","last_fullsized":"2024-03-11T21:49:58.703679+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:40:51.590266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703611+0000","last_change":"2024-03-11T21:49:29.252571+0000","last_active":"2024-03-11T21:49:58.703611+0000","last_peered":"2024-03-11T21:49:58.703611+0000","last_clean":"2024-03-11T21:49:58.703611+0000","last_became_active":"2024-03-11T21:49:29.251938+0000","last_became_peered":"2024-03-11T21:49:29.251938+0000","last_unstale":"2024-03-11T21:49:58.703611+0000","last_undegraded":"2024-03-11T21:49:58.703611+0000","last_fullsized":"2024-03-11T21:49:58.703611+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:24:18.723480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701554+0000","last_change":"2024-03-11T21:49:31.252435+0000","last_active":"2024-03-11T21:49:58.701554+0000","last_peered":"2024-03-11T21:49:58.701554+0000","last_clean":"2024-03-11T21:49:58.701554+0000","last_became_active":"2024-03-11T21:49:31.252055+0000","last_became_peered":"2024-03-11T21:49:31.252055+0000","last_unstale":"2024-03-11T21:49:58.701554+0000","last_undegraded":"2024-03-11T21:49:58.701554+0000","last_fullsized":"2024-03-11T21:49:58.701554+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:33:56.951573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703243+0000","last_change":"2024-03-11T21:49:33.259952+0000","last_active":"2024-03-11T21:49:58.703243+0000","last_peered":"2024-03-11T21:49:58.703243+0000","last_clean":"2024-03-11T21:49:58.703243+0000","last_became_active":"2024-03-11T21:49:33.259268+0000","last_became_peered":"2024-03-11T21:49:33.259268+0000","last_unstale":"2024-03-11T21:49:58.703243+0000","last_undegraded":"2024-03-11T21:49:58.703243+0000","last_fullsized":"2024-03-11T21:49:58.703243+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:26:18.012745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701948+0000","last_change":"2024-03-11T21:49:35.268219+0000","last_active":"2024-03-11T21:49:58.701948+0000","last_peered":"2024-03-11T21:49:58.701948+0000","last_clean":"2024-03-11T21:49:58.701948+0000","last_became_active":"2024-03-11T21:49:35.267860+0000","last_became_peered":"2024-03-11T21:49:35.267860+0000","last_unstale":"2024-03-11T21:49:58.701948+0000","last_undegraded":"2024-03-11T21:49:58.701948+0000","last_fullsized":"2024-03-11T21:49:58.701948+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:30:03.300547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:50:14.307792+0000","last_change":"2024-03-11T21:49:33.260136+0000","last_active":"2024-03-11T21:50:14.307792+0000","last_peered":"2024-03-11T21:50:14.307792+0000","last_clean":"2024-03-11T21:50:14.307792+0000","last_became_active":"2024-03-11T21:49:33.259528+0000","last_became_peered":"2024-03-11T21:49:33.259528+0000","last_unstale":"2024-03-11T21:50:14.307792+0000","last_undegraded":"2024-03-11T21:50:14.307792+0000","last_fullsized":"2024-03-11T21:50:14.307792+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:42:02.181473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704656+0000","last_change":"2024-03-11T21:49:29.254391+0000","last_active":"2024-03-11T21:49:58.704656+0000","last_peered":"2024-03-11T21:49:58.704656+0000","last_clean":"2024-03-11T21:49:58.704656+0000","last_became_active":"2024-03-11T21:49:29.252095+0000","last_became_peered":"2024-03-11T21:49:29.252095+0000","last_unstale":"2024-03-11T21:49:58.704656+0000","last_undegraded":"2024-03-11T21:49:58.704656+0000","last_fullsized":"2024-03-11T21:49:58.704656+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:29:27.311018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'19","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703151+0000","last_change":"2024-03-11T21:49:31.256561+0000","last_active":"2024-03-11T21:49:58.703151+0000","last_peered":"2024-03-11T21:49:58.703151+0000","last_clean":"2024-03-11T21:49:58.703151+0000","last_became_active":"2024-03-11T21:49:31.256225+0000","last_became_peered":"2024-03-11T21:49:31.256225+0000","last_unstale":"2024-03-11T21:49:58.703151+0000","last_undegraded":"2024-03-11T21:49:58.703151+0000","last_fullsized":"2024-03-11T21:49:58.703151+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:14:17.446682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703528+0000","last_change":"2024-03-11T21:49:35.269375+0000","last_active":"2024-03-11T21:49:58.703528+0000","last_peered":"2024-03-11T21:49:58.703528+0000","last_clean":"2024-03-11T21:49:58.703528+0000","last_became_active":"2024-03-11T21:49:35.269052+0000","last_became_peered":"2024-03-11T21:49:35.269052+0000","last_unstale":"2024-03-11T21:49:58.703528+0000","last_undegraded":"2024-03-11T21:49:58.703528+0000","last_fullsized":"2024-03-11T21:49:58.703528+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:04:02.650924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704366+0000","last_change":"2024-03-11T21:49:29.253121+0000","last_active":"2024-03-11T21:49:58.704366+0000","last_peered":"2024-03-11T21:49:58.704366+0000","last_clean":"2024-03-11T21:49:58.704366+0000","last_became_active":"2024-03-11T21:49:29.252761+0000","last_became_peered":"2024-03-11T21:49:29.252761+0000","last_unstale":"2024-03-11T21:49:58.704366+0000","last_undegraded":"2024-03-11T21:49:58.704366+0000","last_fullsized":"2024-03-11T21:49:58.704366+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:07:22.301323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703826+0000","last_change":"2024-03-11T21:49:31.254735+0000","last_active":"2024-03-11T21:49:58.703826+0000","last_peered":"2024-03-11T21:49:58.703826+0000","last_clean":"2024-03-11T21:49:58.703826+0000","last_became_active":"2024-03-11T21:49:31.253920+0000","last_became_peered":"2024-03-11T21:49:31.253920+0000","last_unstale":"2024-03-11T21:49:58.703826+0000","last_undegraded":"2024-03-11T21:49:58.703826+0000","last_fullsized":"2024-03-11T21:49:58.703826+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:46:16.897253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702522+0000","last_change":"2024-03-11T21:49:33.259568+0000","last_active":"2024-03-11T21:49:58.702522+0000","last_peered":"2024-03-11T21:49:58.702522+0000","last_clean":"2024-03-11T21:49:58.702522+0000","last_became_active":"2024-03-11T21:49:33.259214+0000","last_became_peered":"2024-03-11T21:49:33.259214+0000","last_unstale":"2024-03-11T21:49:58.702522+0000","last_undegraded":"2024-03-11T21:49:58.702522+0000","last_fullsized":"2024-03-11T21:49:58.702522+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:54:38.253162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.700536+0000","last_change":"2024-03-11T21:49:35.264133+0000","last_active":"2024-03-11T21:49:58.700536+0000","last_peered":"2024-03-11T21:49:58.700536+0000","last_clean":"2024-03-11T21:49:58.700536+0000","last_became_active":"2024-03-11T21:49:35.263778+0000","last_became_peered":"2024-03-11T21:49:35.263778+0000","last_unstale":"2024-03-11T21:49:58.700536+0000","last_undegraded":"2024-03-11T21:49:58.700536+0000","last_fullsized":"2024-03-11T21:49:58.700536+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:53.404266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.704974+0000","last_change":"2024-03-11T21:49:29.254789+0000","last_active":"2024-03-11T21:49:58.704974+0000","last_peered":"2024-03-11T21:49:58.704974+0000","last_clean":"2024-03-11T21:49:58.704974+0000","last_became_active":"2024-03-11T21:49:29.253502+0000","last_became_peered":"2024-03-11T21:49:29.253502+0000","last_unstale":"2024-03-11T21:49:58.704974+0000","last_undegraded":"2024-03-11T21:49:58.704974+0000","last_fullsized":"2024-03-11T21:49:58.704974+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:16:05.883121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703537+0000","last_change":"2024-03-11T21:49:31.255652+0000","last_active":"2024-03-11T21:49:58.703537+0000","last_peered":"2024-03-11T21:49:58.703537+0000","last_clean":"2024-03-11T21:49:58.703537+0000","last_became_active":"2024-03-11T21:49:31.254989+0000","last_became_peered":"2024-03-11T21:49:31.254989+0000","last_unstale":"2024-03-11T21:49:58.703537+0000","last_undegraded":"2024-03-11T21:49:58.703537+0000","last_fullsized":"2024-03-11T21:49:58.703537+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:34:56.202135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.701041+0000","last_change":"2024-03-11T21:49:33.262443+0000","last_active":"2024-03-11T21:49:58.701041+0000","last_peered":"2024-03-11T21:49:58.701041+0000","last_clean":"2024-03-11T21:49:58.701041+0000","last_became_active":"2024-03-11T21:49:33.262045+0000","last_became_peered":"2024-03-11T21:49:33.262045+0000","last_unstale":"2024-03-11T21:49:58.701041+0000","last_undegraded":"2024-03-11T21:49:58.701041+0000","last_fullsized":"2024-03-11T21:49:58.701041+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:47:04.435813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702159+0000","last_change":"2024-03-11T21:49:35.263806+0000","last_active":"2024-03-11T21:49:58.702159+0000","last_peered":"2024-03-11T21:49:58.702159+0000","last_clean":"2024-03-11T21:49:58.702159+0000","last_became_active":"2024-03-11T21:49:35.263287+0000","last_became_peered":"2024-03-11T21:49:35.263287+0000","last_unstale":"2024-03-11T21:49:58.702159+0000","last_undegraded":"2024-03-11T21:49:58.702159+0000","last_fullsized":"2024-03-11T21:49:58.702159+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.246265+0000","last_clean_scrub_stamp":"2024-03-11T21:49:34.246265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:30:47.721743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.703431+0000","last_change":"2024-03-11T21:49:29.250345+0000","last_active":"2024-03-11T21:49:58.703431+0000","last_peered":"2024-03-11T21:49:58.703431+0000","last_clean":"2024-03-11T21:49:58.703431+0000","last_became_active":"2024-03-11T21:49:29.249600+0000","last_became_peered":"2024-03-11T21:49:29.249600+0000","last_unstale":"2024-03-11T21:49:58.703431+0000","last_undegraded":"2024-03-11T21:49:58.703431+0000","last_fullsized":"2024-03-11T21:49:58.703431+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:28.236672+0000","last_clean_scrub_stamp":"2024-03-11T21:49:28.236672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:09:13.532627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.705327+0000","last_change":"2024-03-11T21:49:31.255212+0000","last_active":"2024-03-11T21:49:58.705327+0000","last_peered":"2024-03-11T21:49:58.705327+0000","last_clean":"2024-03-11T21:49:58.705327+0000","last_became_active":"2024-03-11T21:49:31.254631+0000","last_became_peered":"2024-03-11T21:49:31.254631+0000","last_unstale":"2024-03-11T21:49:58.705327+0000","last_undegraded":"2024-03-11T21:49:58.705327+0000","last_fullsized":"2024-03-11T21:49:58.705327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:30.239752+0000","last_clean_scrub_stamp":"2024-03-11T21:49:30.239752+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:35:43.412087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T21:49:58.702208+0000","last_change":"2024-03-11T21:49:33.259910+0000","last_active":"2024-03-11T21:49:58.702208+0000","last_peered":"2024-03-11T21:49:58.702208+0000","last_clean":"2024-03-11T21:49:58.702208+0000","last_became_active":"2024-03-11T21:49:33.259187+0000","last_became_peered":"2024-03-11T21:49:33.259187+0000","last_unstale":"2024-03-11T21:49:58.702208+0000","last_undegraded":"2024-03-11T21:49:58.702208+0000","last_fullsized":"2024-03-11T21:49:58.702208+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:32.242798+0000","last_clean_scrub_stamp":"2024-03-11T21:49:32.242798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:19:32.487442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411344,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738393,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:50:22.222 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T21:50:22.222 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T21:50:22.222 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T21:50:22.222 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T21:50:22.522 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:22 smithi002 ceph-mon[25764]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:22.635 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:22 smithi165 ceph-mon[27942]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:22.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:22 smithi122 ceph-mon[27585]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:23.360 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:23.519 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:23 smithi002 ceph-mon[25764]: from='client.14511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:50:23.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:23 smithi122 ceph-mon[27585]: from='client.14511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:50:23.783 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:23 smithi165 ceph-mon[27942]: from='client.14511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:50:24.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:24 smithi122 ceph-mon[27585]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:24.475 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:24 smithi002 ceph-mon[25764]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:24.476 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:24 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: ::ffff:172.21.15.122 - - [11/Mar/2024:21:50:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:24.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:24 smithi165 ceph-mon[27942]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:25.267 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T21:50:25.267 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T21:50:25.267 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:25 smithi002 podman[49063]: 2024-03-11 21:50:25.121477567 +0000 UTC m=+0.352519498 volume create a552620b1b899cc5768a0ff887f6787280b32ed43b95ba82bc894a99351c0990 2024-03-11T21:50:25.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:25 smithi122 ceph-mon[27585]: from='client.? 172.21.15.2:0/913180502' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:50:25.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:25 smithi002 ceph-mon[25764]: from='client.? 172.21.15.2:0/913180502' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:50:25.646 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:25 smithi002 podman[49063]: 2024-03-11 21:50:25.407559122 +0000 UTC m=+0.638601051 container create d16e62793684e11801b20d27246bcae26defa639761a068ee3fc886765c44a89 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:50:25.664 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:25 smithi165 ceph-mon[27942]: from='client.? 172.21.15.2:0/913180502' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:50:26.345 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T21:50:26.346 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T21:50:26.346 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T21:50:26.357 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2024-03-11T21:50:26.358 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-11T21:50:26.623 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:26.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:26 smithi002 ceph-mon[25764]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:26.645 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:26 smithi002 podman[49063]: 2024-03-11 21:50:26.446471772 +0000 UTC m=+1.677513723 container init d16e62793684e11801b20d27246bcae26defa639761a068ee3fc886765c44a89 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:50:26.645 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:26 smithi002 podman[49063]: 2024-03-11 21:50:26.452765949 +0000 UTC m=+1.683807880 container start d16e62793684e11801b20d27246bcae26defa639761a068ee3fc886765c44a89 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:50:26.646 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:26 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a[49127]: ts=2024-03-11T21:50:26.465Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-11T21:50:26.646 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:26 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a[49127]: ts=2024-03-11T21:50:26.465Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-11T21:50:26.646 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:26 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a[49127]: ts=2024-03-11T21:50:26.466Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.2 port=9094 2024-03-11T21:50:26.646 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:26 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a[49127]: ts=2024-03-11T21:50:26.467Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T21:50:26.646 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:26 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a[49127]: ts=2024-03-11T21:50:26.497Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T21:50:26.646 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:26 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a[49127]: ts=2024-03-11T21:50:26.498Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T21:50:26.646 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:26 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a[49127]: ts=2024-03-11T21:50:26.500Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-11T21:50:26.646 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:26 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a[49127]: ts=2024-03-11T21:50:26.500Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T21:50:26.646 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:26 smithi002 bash[49063]: d16e62793684e11801b20d27246bcae26defa639761a068ee3fc886765c44a89 2024-03-11T21:50:26.646 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:26 smithi002 systemd[1]: Started Ceph alertmanager.a for 85326acc-dff0-11ee-95c8-87774f69a715. 2024-03-11T21:50:26.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:26 smithi122 ceph-mon[27585]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:26.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:26 smithi165 ceph-mon[27942]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:27.014 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:26 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:26] ENGINE Bus STOPPING 2024-03-11T21:50:27.014 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:26 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:50:27.014 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:26 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:26] ENGINE Bus STOPPED 2024-03-11T21:50:27.014 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:26 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:26] ENGINE Bus STARTING 2024-03-11T21:50:27.014 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:26 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:26] ENGINE Serving on http://:::9283 2024-03-11T21:50:27.015 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:26 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:26] ENGINE Bus STARTED 2024-03-11T21:50:27.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:27 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:27 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:27 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:27 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:27 smithi002 ceph-mon[25764]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:50:27.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:27 smithi002 ceph-mon[25764]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:27.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:27 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:27 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:27 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:50:27.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:27 smithi002 ceph-mon[25764]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:50:27.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:27 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:27 smithi002 ceph-mon[25764]: Deploying daemon grafana.a on smithi122 2024-03-11T21:50:27.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:27 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:27 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.647 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:27 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:27 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:27 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.833 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:27 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:27 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:27 smithi165 ceph-mon[27942]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:50:27.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:27 smithi165 ceph-mon[27942]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:27.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:27 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:27 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:27 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:50:27.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:27 smithi165 ceph-mon[27942]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:50:27.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:27 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:27 smithi165 ceph-mon[27942]: Deploying daemon grafana.a on smithi122 2024-03-11T21:50:27.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:27 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:27 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.834 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:27 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.875 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:27 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.875 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:27 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.875 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:27 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.876 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:27 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.876 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:27 smithi122 ceph-mon[27585]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:50:27.876 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:27 smithi122 ceph-mon[27585]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:27.876 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:27 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.876 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:27 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.876 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:27 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:50:27.876 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:27 smithi122 ceph-mon[27585]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:50:27.876 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:27 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.876 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:27 smithi122 ceph-mon[27585]: Deploying daemon grafana.a on smithi122 2024-03-11T21:50:27.876 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:27 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.876 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:27 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:27.876 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:27 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:28.805 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:28 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a[49127]: ts=2024-03-11T21:50:28.468Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000504991s 2024-03-11T21:50:29.062 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:28 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:29.062 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:28 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:29.062 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:28 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:29.063 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:28 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:29.063 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:28 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:29.167 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:28 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:29.167 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:28 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:29.167 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:28 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:29.168 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:28 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:29.168 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:28 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:29.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:28 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:29.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:28 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:29.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:28 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:29.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:28 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:29.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:28 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:29.958 INFO:teuthology.orchestra.run.smithi002.stdout:167 167 2024-03-11T21:50:30.537 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:30 smithi002 ceph-mon[25764]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:30.537 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:30 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:30.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:30 smithi165 ceph-mon[27942]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:30.585 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:30 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:30.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:30 smithi122 ceph-mon[27585]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:30.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:30 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:31.599 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-11T21:50:32.247 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:32.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:32 smithi002 ceph-mon[25764]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:32.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:32 smithi165 ceph-mon[27942]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:32.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:32 smithi122 ceph-mon[27585]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:33.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:33 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:33.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:33 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:33.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:33 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:34.250 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:34 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: ::ffff:172.21.15.122 - - [11/Mar/2024:21:50:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:34.525 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:34 smithi002 ceph-mon[25764]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:34.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:34 smithi165 ceph-mon[27942]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:34.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:34 smithi122 ceph-mon[27585]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:34.825 INFO:teuthology.orchestra.run.smithi002.stdout:Backend: cephadm 2024-03-11T21:50:34.825 INFO:teuthology.orchestra.run.smithi002.stdout:Available: Yes 2024-03-11T21:50:34.825 INFO:teuthology.orchestra.run.smithi002.stdout:Paused: No 2024-03-11T21:50:35.821 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-11T21:50:36.183 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:36.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:36 smithi002 ceph-mon[25764]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:36.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:36 smithi002 ceph-mon[25764]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:36.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:36 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:36.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:36 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:36.561 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:36 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:36.561 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:50:36 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a[49127]: ts=2024-03-11T21:50:36.469Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002340608s 2024-03-11T21:50:36.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:36 smithi165 ceph-mon[27942]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:36.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:36 smithi165 ceph-mon[27942]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:36.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:36 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:36.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:36 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:36.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:36 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:36.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:36 smithi122 ceph-mon[27585]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:36.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:36 smithi122 ceph-mon[27585]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:36.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:36 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:36.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:36 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:36.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:36 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:38.065 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T21:50:38.066 INFO:teuthology.orchestra.run.smithi002.stdout:agent.smithi002 smithi002 running 1s ago 3m - - 2024-03-11T21:50:38.066 INFO:teuthology.orchestra.run.smithi002.stdout:agent.smithi122 smithi122 running 10s ago 3m - - 2024-03-11T21:50:38.066 INFO:teuthology.orchestra.run.smithi002.stdout:agent.smithi165 smithi165 running 11s ago 3m - - 2024-03-11T21:50:38.066 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (13s) 1s ago 11s 15.8M - 0.25.0 c8568f914cd2 d16e62793684 2024-03-11T21:50:38.066 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi122 *:8443,8765 running (2m) 10s ago 2m 448M - 19.0.0-1607-g94a53970 a6477a474d71 31ff6547d310 2024-03-11T21:50:38.066 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:9283,8765 running (4m) 1s ago 4m 499M - 19.0.0-1607-g94a53970 a6477a474d71 0aba717a9141 2024-03-11T21:50:38.066 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.z smithi165 *:8443,8765 running (2m) 11s ago 2m 448M - 19.0.0-1607-g94a53970 a6477a474d71 528f350cb9e2 2024-03-11T21:50:38.066 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (4m) 1s ago 4m 45.9M 2048M 19.0.0-1607-g94a53970 a6477a474d71 64d5c323ec3b 2024-03-11T21:50:38.066 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi122 running (3m) 10s ago 2m 36.9M 2048M 19.0.0-1607-g94a53970 a6477a474d71 40175a808c6d 2024-03-11T21:50:38.066 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi165 running (3m) 11s ago 3m 36.4M 2048M 19.0.0-1607-g94a53970 a6477a474d71 ad1ae1a88930 2024-03-11T21:50:38.066 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (2m) 1s ago 2m 51.5M 4096M 19.0.0-1607-g94a53970 a6477a474d71 074468c85393 2024-03-11T21:50:38.066 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi122 running (116s) 10s ago 115s 59.7M 4096M 19.0.0-1607-g94a53970 a6477a474d71 ecdbfc322e36 2024-03-11T21:50:38.067 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi165 running (89s) 11s ago 89s 53.3M 4096M 19.0.0-1607-g94a53970 a6477a474d71 ed0cc7a46dff 2024-03-11T21:50:38.067 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi122 *:9095 running (49s) 10s ago 47s 35.6M - 2.43.0 a07b618ecd1d 0becd75e25cc 2024-03-11T21:50:38.067 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.a smithi002 *:80 running (70s) 1s ago 70s 76.9M - 19.0.0-1607-g94a53970 a6477a474d71 9837194d9807 2024-03-11T21:50:38.361 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:38 smithi002 ceph-mon[25764]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:38.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:38 smithi165 ceph-mon[27942]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:38.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:38 smithi122 ceph-mon[27585]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:38.758 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-11T21:50:39.002 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:39.373 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:39 smithi002 ceph-mon[25764]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:39.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:39 smithi165 ceph-mon[27942]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:39.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:39 smithi122 ceph-mon[27585]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:40.482 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:40 smithi002 ceph-mon[25764]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:40.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:40 smithi165 ceph-mon[27942]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:40.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:40 smithi122 ceph-mon[27585]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:40.962 INFO:teuthology.orchestra.run.smithi002.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-11T21:50:40.963 INFO:teuthology.orchestra.run.smithi002.stdout:agent 3/3 14s ago 42s * 2024-03-11T21:50:40.963 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager ?:9093,9094 1/1 4s ago 68s smithi002=a;count:1 2024-03-11T21:50:40.963 INFO:teuthology.orchestra.run.smithi002.stdout:grafana ?:3000 0/1 - 65s smithi122=a;count:1 2024-03-11T21:50:40.963 INFO:teuthology.orchestra.run.smithi002.stdout:mgr 3/3 14s ago 2m smithi002=y;smithi122=x;smithi165=z;count:3 2024-03-11T21:50:40.963 INFO:teuthology.orchestra.run.smithi002.stdout:mon 3/3 14s ago 3m smithi002:172.21.15.2=a;smithi122:172.21.15.122=b;smithi165:172.21.15.165=c;count:3 2024-03-11T21:50:40.963 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter ?:9100 0/3 - 70s smithi002=a;smithi122=b;smithi165=c;count:3 2024-03-11T21:50:40.963 INFO:teuthology.orchestra.run.smithi002.stdout:osd 3 14s ago - 2024-03-11T21:50:40.963 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus ?:9095 1/1 13s ago 73s smithi122=a;count:1 2024-03-11T21:50:40.963 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.a ?:80 1/1 4s ago 73s smithi002=foo.a;count:1 2024-03-11T21:50:41.546 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-11T21:50:41.789 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:42.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:42 smithi002 ceph-mon[25764]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:42.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:42 smithi002 ceph-mon[25764]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:42.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:42 smithi165 ceph-mon[27942]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:42.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:42 smithi165 ceph-mon[27942]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:42.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:42 smithi122 ceph-mon[27585]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:42.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:42 smithi122 ceph-mon[27585]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:44.255 INFO:teuthology.orchestra.run.smithi002.stdout:HOST ADDR LABELS STATUS 2024-03-11T21:50:44.255 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 172.21.15.2 2024-03-11T21:50:44.255 INFO:teuthology.orchestra.run.smithi002.stdout:smithi122 172.21.15.122 2024-03-11T21:50:44.255 INFO:teuthology.orchestra.run.smithi002.stdout:smithi165 172.21.15.165 2024-03-11T21:50:44.255 INFO:teuthology.orchestra.run.smithi002.stdout:3 hosts in cluster 2024-03-11T21:50:44.542 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:44 smithi002 ceph-mon[25764]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:44.542 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:44 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:44.543 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:44 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: ::ffff:172.21.15.122 - - [11/Mar/2024:21:50:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:44.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:44 smithi165 ceph-mon[27942]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:44.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:44 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:44.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:44 smithi122 ceph-mon[27585]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:44.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:44 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:45.517 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-11T21:50:45.542 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:45 smithi002 ceph-mon[25764]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:45.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:45 smithi165 ceph-mon[27942]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:45.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:45 smithi122 ceph-mon[27585]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:45.844 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:46.528 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:46 smithi002 ceph-mon[25764]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:46.583 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:46 smithi165 ceph-mon[27942]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:46.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:46 smithi122 ceph-mon[27585]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:48.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:48 smithi165 ceph-mon[27942]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:48.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:48 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:48 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:48 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:48 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:48 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:48 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-11T21:50:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300EV480DGN 447G No 12s ago Has a FileSystem, LVM detected 2024-03-11T21:50:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 /dev/nvme1n1 ssd Linux_f7f8e8ac88e01da4278c 89.4G Yes 12s ago 2024-03-11T21:50:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 /dev/nvme2n1 ssd Linux_71f6e8e759e933b228ad 89.4G Yes 12s ago 2024-03-11T21:50:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 /dev/nvme3n1 ssd Linux_b530331a0339789641d7 89.4G Yes 12s ago 2024-03-11T21:50:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 /dev/nvme4n1 ssd Linux_13ee583bb841c752b78a 89.4G No 12s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T21:50:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W4HPVS 931G No 12s ago Has partitions 2024-03-11T21:50:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:smithi122 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010U400BGN 372G No 0s ago Has a FileSystem, LVM detected 2024-03-11T21:50:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:smithi122 /dev/nvme1n1 ssd Linux_712b7a166060c9b8f5c9 89.4G Yes 0s ago 2024-03-11T21:50:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:smithi122 /dev/nvme2n1 ssd Linux_efbeb717c8f073555ee3 89.4G Yes 0s ago 2024-03-11T21:50:48.390 INFO:teuthology.orchestra.run.smithi002.stdout:smithi122 /dev/nvme3n1 ssd Linux_738fefb58a792da8d2fc 89.4G Yes 0s ago 2024-03-11T21:50:48.390 INFO:teuthology.orchestra.run.smithi002.stdout:smithi122 /dev/nvme4n1 ssd Linux_3a03a60bebc646033a82 89.4G No 0s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T21:50:48.390 INFO:teuthology.orchestra.run.smithi002.stdout:smithi122 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5L5J2 931G No 0s ago Has partitions 2024-03-11T21:50:48.390 INFO:teuthology.orchestra.run.smithi002.stdout:smithi165 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300CE400BGN 372G No 1s ago Has a FileSystem, LVM detected 2024-03-11T21:50:48.390 INFO:teuthology.orchestra.run.smithi002.stdout:smithi165 /dev/nvme1n1 ssd Linux_0b67468dd008f9ae3921 89.4G Yes 1s ago 2024-03-11T21:50:48.390 INFO:teuthology.orchestra.run.smithi002.stdout:smithi165 /dev/nvme2n1 ssd Linux_2f621fa837f0ee352d1d 89.4G Yes 1s ago 2024-03-11T21:50:48.390 INFO:teuthology.orchestra.run.smithi002.stdout:smithi165 /dev/nvme3n1 ssd Linux_c2cf2f321488040ebe03 89.4G Yes 1s ago 2024-03-11T21:50:48.390 INFO:teuthology.orchestra.run.smithi002.stdout:smithi165 /dev/nvme4n1 ssd Linux_517b49c11fe5ca538c85 89.4G No 1s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T21:50:48.390 INFO:teuthology.orchestra.run.smithi002.stdout:smithi165 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XG09 931G No 1s ago Has partitions 2024-03-11T21:50:48.390 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:48 smithi002 ceph-mon[25764]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:48.391 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:48 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.391 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:48 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.391 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:48 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.391 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:48 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.391 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:48 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.391 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:48 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:48 smithi122 ceph-mon[27585]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:48.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:48 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:48 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:48 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:48 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:48 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:48.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:48 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:49.004 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-11T21:50:49.291 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:49.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:49 smithi165 ceph-mon[27942]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:49.392 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:49 smithi002 ceph-mon[25764]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:49.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:49 smithi122 ceph-mon[27585]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:50.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:50 smithi165 ceph-mon[27942]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:50.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:50 smithi002 ceph-mon[25764]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:50.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:50 smithi122 ceph-mon[27585]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:51.133 INFO:teuthology.orchestra.run.smithi002.stdout:service_type: agent 2024-03-11T21:50:51.134 INFO:teuthology.orchestra.run.smithi002.stdout:service_name: agent 2024-03-11T21:50:51.134 INFO:teuthology.orchestra.run.smithi002.stdout:placement: 2024-03-11T21:50:51.134 INFO:teuthology.orchestra.run.smithi002.stdout: host_pattern: '*' 2024-03-11T21:50:51.134 INFO:teuthology.orchestra.run.smithi002.stdout:status: 2024-03-11T21:50:51.134 INFO:teuthology.orchestra.run.smithi002.stdout: created: '2024-03-11T21:49:58.758218Z' 2024-03-11T21:50:51.134 INFO:teuthology.orchestra.run.smithi002.stdout: last_refresh: '2024-03-11T21:50:47.059964Z' 2024-03-11T21:50:51.134 INFO:teuthology.orchestra.run.smithi002.stdout: running: 3 2024-03-11T21:50:51.134 INFO:teuthology.orchestra.run.smithi002.stdout: size: 3 2024-03-11T21:50:51.134 INFO:teuthology.orchestra.run.smithi002.stdout:events: 2024-03-11T21:50:51.134 INFO:teuthology.orchestra.run.smithi002.stdout:- 2024-03-11T21:50:00.212763Z service:agent [INFO] "service was created" 2024-03-11T21:50:51.134 INFO:teuthology.orchestra.run.smithi002.stdout:--- 2024-03-11T21:50:51.134 INFO:teuthology.orchestra.run.smithi002.stdout:service_type: alertmanager 2024-03-11T21:50:51.135 INFO:teuthology.orchestra.run.smithi002.stdout:service_name: alertmanager 2024-03-11T21:50:51.135 INFO:teuthology.orchestra.run.smithi002.stdout:placement: 2024-03-11T21:50:51.135 INFO:teuthology.orchestra.run.smithi002.stdout: count: 1 2024-03-11T21:50:51.135 INFO:teuthology.orchestra.run.smithi002.stdout: hosts: 2024-03-11T21:50:51.135 INFO:teuthology.orchestra.run.smithi002.stdout: - smithi002=a 2024-03-11T21:50:51.135 INFO:teuthology.orchestra.run.smithi002.stdout:status: 2024-03-11T21:50:51.135 INFO:teuthology.orchestra.run.smithi002.stdout: created: '2024-03-11T21:49:32.694752Z' 2024-03-11T21:50:51.135 INFO:teuthology.orchestra.run.smithi002.stdout: last_refresh: '2024-03-11T21:50:50.894174Z' 2024-03-11T21:50:51.135 INFO:teuthology.orchestra.run.smithi002.stdout: ports: 2024-03-11T21:50:51.135 INFO:teuthology.orchestra.run.smithi002.stdout: - 9093 2024-03-11T21:50:51.135 INFO:teuthology.orchestra.run.smithi002.stdout: - 9094 2024-03-11T21:50:51.135 INFO:teuthology.orchestra.run.smithi002.stdout: running: 1 2024-03-11T21:50:51.135 INFO:teuthology.orchestra.run.smithi002.stdout: size: 1 2024-03-11T21:50:51.135 INFO:teuthology.orchestra.run.smithi002.stdout:events: 2024-03-11T21:50:51.136 INFO:teuthology.orchestra.run.smithi002.stdout:- 2024-03-11T21:50:26.646739Z service:alertmanager [INFO] "service was created" 2024-03-11T21:50:51.136 INFO:teuthology.orchestra.run.smithi002.stdout:--- 2024-03-11T21:50:51.136 INFO:teuthology.orchestra.run.smithi002.stdout:service_type: grafana 2024-03-11T21:50:51.136 INFO:teuthology.orchestra.run.smithi002.stdout:service_name: grafana 2024-03-11T21:50:51.136 INFO:teuthology.orchestra.run.smithi002.stdout:placement: 2024-03-11T21:50:51.136 INFO:teuthology.orchestra.run.smithi002.stdout: count: 1 2024-03-11T21:50:51.136 INFO:teuthology.orchestra.run.smithi002.stdout: hosts: 2024-03-11T21:50:51.136 INFO:teuthology.orchestra.run.smithi002.stdout: - smithi122=a 2024-03-11T21:50:51.136 INFO:teuthology.orchestra.run.smithi002.stdout:spec: 2024-03-11T21:50:51.136 INFO:teuthology.orchestra.run.smithi002.stdout: anonymous_access: true 2024-03-11T21:50:51.136 INFO:teuthology.orchestra.run.smithi002.stdout: protocol: https 2024-03-11T21:50:51.136 INFO:teuthology.orchestra.run.smithi002.stdout:status: 2024-03-11T21:50:51.136 INFO:teuthology.orchestra.run.smithi002.stdout: created: '2024-03-11T21:49:35.362863Z' 2024-03-11T21:50:51.136 INFO:teuthology.orchestra.run.smithi002.stdout: ports: 2024-03-11T21:50:51.136 INFO:teuthology.orchestra.run.smithi002.stdout: - 3000 2024-03-11T21:50:51.137 INFO:teuthology.orchestra.run.smithi002.stdout: running: 0 2024-03-11T21:50:51.137 INFO:teuthology.orchestra.run.smithi002.stdout: size: 1 2024-03-11T21:50:51.137 INFO:teuthology.orchestra.run.smithi002.stdout:--- 2024-03-11T21:50:51.137 INFO:teuthology.orchestra.run.smithi002.stdout:service_type: mgr 2024-03-11T21:50:51.137 INFO:teuthology.orchestra.run.smithi002.stdout:service_name: mgr 2024-03-11T21:50:51.137 INFO:teuthology.orchestra.run.smithi002.stdout:placement: 2024-03-11T21:50:51.137 INFO:teuthology.orchestra.run.smithi002.stdout: count: 3 2024-03-11T21:50:51.137 INFO:teuthology.orchestra.run.smithi002.stdout: hosts: 2024-03-11T21:50:51.137 INFO:teuthology.orchestra.run.smithi002.stdout: - smithi002=y 2024-03-11T21:50:51.137 INFO:teuthology.orchestra.run.smithi002.stdout: - smithi122=x 2024-03-11T21:50:51.137 INFO:teuthology.orchestra.run.smithi002.stdout: - smithi165=z 2024-03-11T21:50:51.137 INFO:teuthology.orchestra.run.smithi002.stdout:status: 2024-03-11T21:50:51.137 INFO:teuthology.orchestra.run.smithi002.stdout: created: '2024-03-11T21:47:50.844130Z' 2024-03-11T21:50:51.137 INFO:teuthology.orchestra.run.smithi002.stdout: last_refresh: '2024-03-11T21:50:47.060179Z' 2024-03-11T21:50:51.137 INFO:teuthology.orchestra.run.smithi002.stdout: running: 3 2024-03-11T21:50:51.138 INFO:teuthology.orchestra.run.smithi002.stdout: size: 3 2024-03-11T21:50:51.138 INFO:teuthology.orchestra.run.smithi002.stdout:--- 2024-03-11T21:50:51.138 INFO:teuthology.orchestra.run.smithi002.stdout:service_type: mon 2024-03-11T21:50:51.138 INFO:teuthology.orchestra.run.smithi002.stdout:service_name: mon 2024-03-11T21:50:51.138 INFO:teuthology.orchestra.run.smithi002.stdout:placement: 2024-03-11T21:50:51.138 INFO:teuthology.orchestra.run.smithi002.stdout: count: 3 2024-03-11T21:50:51.138 INFO:teuthology.orchestra.run.smithi002.stdout: hosts: 2024-03-11T21:50:51.138 INFO:teuthology.orchestra.run.smithi002.stdout: - smithi002:172.21.15.2=a 2024-03-11T21:50:51.138 INFO:teuthology.orchestra.run.smithi002.stdout: - smithi122:172.21.15.122=b 2024-03-11T21:50:51.138 INFO:teuthology.orchestra.run.smithi002.stdout: - smithi165:172.21.15.165=c 2024-03-11T21:50:51.138 INFO:teuthology.orchestra.run.smithi002.stdout:status: 2024-03-11T21:50:51.138 INFO:teuthology.orchestra.run.smithi002.stdout: created: '2024-03-11T21:47:24.386568Z' 2024-03-11T21:50:51.138 INFO:teuthology.orchestra.run.smithi002.stdout: last_refresh: '2024-03-11T21:50:47.060071Z' 2024-03-11T21:50:51.138 INFO:teuthology.orchestra.run.smithi002.stdout: running: 3 2024-03-11T21:50:51.139 INFO:teuthology.orchestra.run.smithi002.stdout: size: 3 2024-03-11T21:50:51.139 INFO:teuthology.orchestra.run.smithi002.stdout:--- 2024-03-11T21:50:51.139 INFO:teuthology.orchestra.run.smithi002.stdout:service_type: node-exporter 2024-03-11T21:50:51.139 INFO:teuthology.orchestra.run.smithi002.stdout:service_name: node-exporter 2024-03-11T21:50:51.139 INFO:teuthology.orchestra.run.smithi002.stdout:placement: 2024-03-11T21:50:51.139 INFO:teuthology.orchestra.run.smithi002.stdout: count: 3 2024-03-11T21:50:51.139 INFO:teuthology.orchestra.run.smithi002.stdout: hosts: 2024-03-11T21:50:51.139 INFO:teuthology.orchestra.run.smithi002.stdout: - smithi002=a 2024-03-11T21:50:51.139 INFO:teuthology.orchestra.run.smithi002.stdout: - smithi122=b 2024-03-11T21:50:51.139 INFO:teuthology.orchestra.run.smithi002.stdout: - smithi165=c 2024-03-11T21:50:51.139 INFO:teuthology.orchestra.run.smithi002.stdout:status: 2024-03-11T21:50:51.139 INFO:teuthology.orchestra.run.smithi002.stdout: created: '2024-03-11T21:49:30.028331Z' 2024-03-11T21:50:51.139 INFO:teuthology.orchestra.run.smithi002.stdout: ports: 2024-03-11T21:50:51.139 INFO:teuthology.orchestra.run.smithi002.stdout: - 9100 2024-03-11T21:50:51.139 INFO:teuthology.orchestra.run.smithi002.stdout: running: 0 2024-03-11T21:50:51.140 INFO:teuthology.orchestra.run.smithi002.stdout: size: 3 2024-03-11T21:50:51.140 INFO:teuthology.orchestra.run.smithi002.stdout:--- 2024-03-11T21:50:51.140 INFO:teuthology.orchestra.run.smithi002.stdout:service_type: osd 2024-03-11T21:50:51.140 INFO:teuthology.orchestra.run.smithi002.stdout:service_name: osd 2024-03-11T21:50:51.140 INFO:teuthology.orchestra.run.smithi002.stdout:unmanaged: true 2024-03-11T21:50:51.140 INFO:teuthology.orchestra.run.smithi002.stdout:spec: 2024-03-11T21:50:51.140 INFO:teuthology.orchestra.run.smithi002.stdout: filter_logic: AND 2024-03-11T21:50:51.140 INFO:teuthology.orchestra.run.smithi002.stdout: objectstore: bluestore 2024-03-11T21:50:51.140 INFO:teuthology.orchestra.run.smithi002.stdout:status: 2024-03-11T21:50:51.140 INFO:teuthology.orchestra.run.smithi002.stdout: container_image_id: a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T21:50:51.140 INFO:teuthology.orchestra.run.smithi002.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:50:51.140 INFO:teuthology.orchestra.run.smithi002.stdout: last_refresh: '2024-03-11T21:50:47.060263Z' 2024-03-11T21:50:51.140 INFO:teuthology.orchestra.run.smithi002.stdout: running: 3 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout: size: 3 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout:--- 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout:service_type: prometheus 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout:service_name: prometheus 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout:placement: 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout: count: 1 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout: hosts: 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout: - smithi122=a 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout:status: 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout: created: '2024-03-11T21:49:27.222638Z' 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout: last_refresh: '2024-03-11T21:50:48.017276Z' 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout: ports: 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout: - 9095 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout: running: 1 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout: size: 1 2024-03-11T21:50:51.141 INFO:teuthology.orchestra.run.smithi002.stdout:--- 2024-03-11T21:50:51.142 INFO:teuthology.orchestra.run.smithi002.stdout:service_type: rgw 2024-03-11T21:50:51.142 INFO:teuthology.orchestra.run.smithi002.stdout:service_id: foo.a 2024-03-11T21:50:51.142 INFO:teuthology.orchestra.run.smithi002.stdout:service_name: rgw.foo.a 2024-03-11T21:50:51.142 INFO:teuthology.orchestra.run.smithi002.stdout:placement: 2024-03-11T21:50:51.142 INFO:teuthology.orchestra.run.smithi002.stdout: count: 1 2024-03-11T21:50:51.142 INFO:teuthology.orchestra.run.smithi002.stdout: hosts: 2024-03-11T21:50:51.142 INFO:teuthology.orchestra.run.smithi002.stdout: - smithi002=foo.a 2024-03-11T21:50:51.142 INFO:teuthology.orchestra.run.smithi002.stdout:status: 2024-03-11T21:50:51.142 INFO:teuthology.orchestra.run.smithi002.stdout: created: '2024-03-11T21:49:27.743544Z' 2024-03-11T21:50:51.142 INFO:teuthology.orchestra.run.smithi002.stdout: last_refresh: '2024-03-11T21:50:50.894081Z' 2024-03-11T21:50:51.142 INFO:teuthology.orchestra.run.smithi002.stdout: ports: 2024-03-11T21:50:51.142 INFO:teuthology.orchestra.run.smithi002.stdout: - 80 2024-03-11T21:50:51.142 INFO:teuthology.orchestra.run.smithi002.stdout: running: 1 2024-03-11T21:50:51.142 INFO:teuthology.orchestra.run.smithi002.stdout: size: 1 2024-03-11T21:50:51.887 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 85326acc-dff0-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-11T21:50:52.137 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:52.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:51 smithi002 ceph-mon[25764]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:52.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:51 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:52.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:51 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:52.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:51 smithi002 ceph-mon[25764]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:52.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:51 smithi002 ceph-mon[25764]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T21:50:52.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:51 smithi122 ceph-mon[27585]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:52.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:51 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:52.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:51 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:52.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:51 smithi122 ceph-mon[27585]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:52.175 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:51 smithi122 ceph-mon[27585]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T21:50:52.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:51 smithi165 ceph-mon[27942]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:52.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:51 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:52.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:51 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:52.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:51 smithi165 ceph-mon[27942]: from='mgr.14421 172.21.15.2:0/1053938960' entity='mgr.y' 2024-03-11T21:50:52.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:51 smithi165 ceph-mon[27942]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T21:50:53.964 INFO:teuthology.orchestra.run.smithi002.stdout:osd 3 6s ago - 2024-03-11T21:50:54.145 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:53 smithi002 ceph-mon[25764]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:54.174 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:53 smithi122 ceph-mon[27585]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:54.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:53 smithi165 ceph-mon[27942]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:50:54.575 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:54 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: ::ffff:172.21.15.122 - - [11/Mar/2024:21:50:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:54.602 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T21:50:54.611 INFO:tasks.cephadm:Teardown begin 2024-03-11T21:50:54.611 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:54.641 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:54.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:50:54.711 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T21:50:54.711 DEBUG:teuthology.orchestra.run.smithi002:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:50:54.726 DEBUG:teuthology.orchestra.run.smithi122:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:50:54.741 DEBUG:teuthology.orchestra.run.smithi165:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:50:54.771 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T21:50:54.771 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T21:50:54.772 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-85326acc-dff0-11ee-95c8-87774f69a715@mon.a 2024-03-11T21:50:54.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:54 smithi002 systemd[1]: Stopping Ceph mon.a for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:50:54.924 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:54 smithi122 ceph-mon[27585]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:55.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:54 smithi165 ceph-mon[27942]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:55.193 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:54 smithi002 ceph-mon[25764]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:50:55.193 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a[25741]: 2024-03-11T21:50:55.080+0000 7fcb111e8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:50:55.193 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a[25741]: 2024-03-11T21:50:55.080+0000 7fcb111e8700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T21:50:55.193 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:55] ENGINE Bus STOPPING 2024-03-11T21:50:55.193 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:50:55.193 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:55] ENGINE Bus STOPPED 2024-03-11T21:50:55.193 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:55] ENGINE Bus STARTING 2024-03-11T21:50:55.537 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:55 smithi002 podman[51879]: 2024-03-11 21:50:55.264716529 +0000 UTC m=+0.300602257 container died 64d5c323ec3ba1e2601d372ddbbdd17ae8dda1095df3a2e2cc52056fc0ace318 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-11T21:50:55.537 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:55] ENGINE Serving on http://:::9283 2024-03-11T21:50:55.537 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:55 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:55] ENGINE Bus STARTED 2024-03-11T21:50:55.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:55 smithi002 podman[51879]: 2024-03-11 21:50:55.736190205 +0000 UTC m=+0.772075934 container cleanup 64d5c323ec3ba1e2601d372ddbbdd17ae8dda1095df3a2e2cc52056fc0ace318 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-11T21:50:55.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 21:50:55 smithi002 bash[51879]: ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-a 2024-03-11T21:50:56.149 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T21:50:56.192 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:50:56.192 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T21:50:56.192 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-11T21:50:56.193 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-85326acc-dff0-11ee-95c8-87774f69a715@mon.b 2024-03-11T21:50:56.372 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:56 smithi122 systemd[1]: Stopping Ceph mon.b for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:50:57.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:57 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-b[27562]: 2024-03-11T21:50:57.404+0000 7f890cd4d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:50:57.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:57 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-b[27562]: 2024-03-11T21:50:57.404+0000 7f890cd4d700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T21:50:57.896 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:57 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:57] ENGINE Bus STOPPING 2024-03-11T21:50:57.896 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:57 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:50:57.896 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:57 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:57] ENGINE Bus STOPPED 2024-03-11T21:50:57.896 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:57 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:57] ENGINE Bus STARTING 2024-03-11T21:50:57.896 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:57 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:57] ENGINE Serving on http://:::9283 2024-03-11T21:50:57.896 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:50:57 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y[26023]: [11/Mar/2024:21:50:57] ENGINE Bus STARTED 2024-03-11T21:50:58.414 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:58 smithi122 podman[38717]: 2024-03-11 21:50:58.167827511 +0000 UTC m=+1.425996268 container died 40175a808c6d8033f168b70b85cec240b8f7c653ca95718a2db5774e3d620af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:50:58.674 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:58 smithi122 podman[38717]: 2024-03-11 21:50:58.582063241 +0000 UTC m=+1.840232008 container cleanup 40175a808c6d8033f168b70b85cec240b8f7c653ca95718a2db5774e3d620af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-11T21:50:58.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:58 smithi122 bash[38717]: ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-b 2024-03-11T21:50:59.076 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 11 21:50:58 smithi122 podman[38781]: 2024-03-11 21:50:58.783513621 +0000 UTC m=+1.242420930 container remove 40175a808c6d8033f168b70b85cec240b8f7c653ca95718a2db5774e3d620af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-b, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-11T21:50:59.076 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 podman[38957]: 2024-03-11 21:50:58.911954906 +0000 UTC m=+0.021286872 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-11T21:50:59.082 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T21:50:59.151 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:50:59.151 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-11T21:50:59.151 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-11T21:50:59.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-85326acc-dff0-11ee-95c8-87774f69a715@mon.c 2024-03-11T21:50:59.337 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 podman[38957]: 2024-03-11 21:50:59.238368205 +0000 UTC m=+0.347700174 container init c7386cf0e58091466c1cef980c93b998cbff1266bc82df0f566d4f572bd6937c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.32.0, description=Ceph Grafana Container, distribution-scope=public, io.openshift.tags=base rhel8, architecture=x86_64, vendor=Red Hat, Inc., name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-03-11T21:50:59.337 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 podman[38957]: 2024-03-11 21:50:59.244571368 +0000 UTC m=+0.353903333 container start c7386cf0e58091466c1cef980c93b998cbff1266bc82df0f566d4f572bd6937c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, release=239.1651231664, version=8.5, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc., name=ubi8, maintainer=Paul Cuzner , io.buildah.version=1.32.0, io.openshift.tags=base rhel8, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-11T21:50:59.481 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:59 smithi165 systemd[1]: Stopping Ceph mon.c for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:50:59.589 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 bash[38957]: c7386cf0e58091466c1cef980c93b998cbff1266bc82df0f566d4f572bd6937c 2024-03-11T21:50:59.589 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 systemd[1]: Started Ceph grafana.a for 85326acc-dff0-11ee-95c8-87774f69a715. 2024-03-11T21:50:59.589 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=settings t=2024-03-11T21:50:59.40381678Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-11T21:50:59.589 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=settings t=2024-03-11T21:50:59.404024516Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-11T21:50:59.590 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=settings t=2024-03-11T21:50:59.404045526Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-11T21:50:59.590 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=settings t=2024-03-11T21:50:59.404052658Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-11T21:50:59.590 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=settings t=2024-03-11T21:50:59.404059386Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-11T21:50:59.590 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=settings t=2024-03-11T21:50:59.404065844Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-11T21:50:59.590 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=settings t=2024-03-11T21:50:59.40407155Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-11T21:50:59.590 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=settings t=2024-03-11T21:50:59.404077308Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-11T21:50:59.590 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=settings t=2024-03-11T21:50:59.404083184Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-11T21:50:59.590 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=settings t=2024-03-11T21:50:59.404088806Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-11T21:50:59.590 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=settings t=2024-03-11T21:50:59.404094144Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-11T21:50:59.590 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=settings t=2024-03-11T21:50:59.404099469Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-11T21:50:59.590 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=settings t=2024-03-11T21:50:59.404105148Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-11T21:50:59.590 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=settings t=2024-03-11T21:50:59.404110827Z level=info msg="App mode production" 2024-03-11T21:50:59.591 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=sqlstore t=2024-03-11T21:50:59.404164768Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-11T21:50:59.591 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=sqlstore t=2024-03-11T21:50:59.404182327Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-11T21:50:59.591 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.405039285Z level=info msg="Starting DB migrations" 2024-03-11T21:50:59.591 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.405121165Z level=info msg="Executing migration" id="create migration_log table" 2024-03-11T21:50:59.591 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.40565304Z level=info msg="Executing migration" id="create user table" 2024-03-11T21:50:59.591 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.406093826Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-11T21:50:59.591 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.408657498Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-11T21:50:59.591 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.41104476Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-11T21:50:59.591 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.413463538Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-11T21:50:59.591 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.413850334Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-11T21:50:59.591 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.414575266Z level=info msg="Executing migration" id="create user table v2" 2024-03-11T21:50:59.591 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.41495985Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-11T21:50:59.592 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.415345429Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-11T21:50:59.592 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.41575464Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-11T21:50:59.592 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.416111362Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-11T21:50:59.592 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.416450247Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-11T21:50:59.592 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.416826743Z level=info msg="Executing migration" id="Update user table charset" 2024-03-11T21:50:59.592 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.417197651Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-11T21:50:59.592 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.417687868Z level=info msg="Executing migration" id="Add missing user data" 2024-03-11T21:50:59.592 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.418070801Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-11T21:50:59.592 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.41851826Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-11T21:50:59.592 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.418946764Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-11T21:50:59.592 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.419444697Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-11T21:50:59.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.42078769Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-11T21:50:59.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.421253106Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-11T21:50:59.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.421691831Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-11T21:50:59.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.422116321Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-11T21:50:59.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.422643381Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-11T21:50:59.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.423176037Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-11T21:50:59.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.423559626Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-11T21:50:59.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.424005338Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-11T21:50:59.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.424561999Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-11T21:50:59.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.425099788Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-11T21:50:59.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.425628694Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-11T21:50:59.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.426596331Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-11T21:50:59.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.427156453Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-11T21:50:59.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.427699507Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-11T21:50:59.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.428259163Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-11T21:50:59.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.428795431Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-11T21:50:59.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.429349739Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-11T21:50:59.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.429804723Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-11T21:50:59.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.430293284Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-11T21:50:59.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.430740904Z level=info msg="Executing migration" id="create star table" 2024-03-11T21:50:59.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.431240662Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-11T21:50:59.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.431754351Z level=info msg="Executing migration" id="create org table v1" 2024-03-11T21:50:59.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.43225339Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-11T21:50:59.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.433008429Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-11T21:50:59.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.433494754Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-11T21:50:59.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.434056909Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-11T21:50:59.596 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.434617319Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-11T21:50:59.596 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.435228705Z level=info msg="Executing migration" id="Update org table charset" 2024-03-11T21:50:59.596 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.436032783Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-11T21:50:59.596 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.436811202Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-11T21:50:59.596 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.439414592Z level=info msg="Executing migration" id="create dashboard table" 2024-03-11T21:50:59.596 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.44187039Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-11T21:50:59.596 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.444471782Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-11T21:50:59.596 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.447149499Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-11T21:50:59.596 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.449775571Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-11T21:50:59.596 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.452244058Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-11T21:50:59.596 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.454737687Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-11T21:50:59.596 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.457446034Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-11T21:50:59.596 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.460139676Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-11T21:50:59.596 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.462813469Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.465462227Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.468008816Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.470540996Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.473270948Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.47604475Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.478692226Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.481467541Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.484021278Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.486434673Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.488903033Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.491590699Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.494305537Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.496904842Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.497414718Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.499028871Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-11T21:50:59.597 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.49953345Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-11T21:50:59.598 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.501662528Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-11T21:50:59.598 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.504379882Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-11T21:50:59.598 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.506974399Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-11T21:50:59.598 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.509412673Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-11T21:50:59.598 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.511826677Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-11T21:50:59.598 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.514276018Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-11T21:50:59.598 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.516731577Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-11T21:50:59.598 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.519185682Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-11T21:50:59.598 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.521732456Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-11T21:50:59.598 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.524162549Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-11T21:50:59.598 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.524948524Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-11T21:50:59.598 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.525710137Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-11T21:50:59.598 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.526497734Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-11T21:50:59.599 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.527287353Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-11T21:50:59.599 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.528107881Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-11T21:50:59.599 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.528895849Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-11T21:50:59.599 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.529674988Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-11T21:50:59.599 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.530472112Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-11T21:50:59.599 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.531265213Z level=info msg="Executing migration" id="create data_source table" 2024-03-11T21:50:59.599 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.532058758Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-11T21:50:59.599 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.532804034Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-11T21:50:59.599 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.533238441Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-11T21:50:59.599 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.534000005Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-11T21:50:59.600 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.534773388Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-11T21:50:59.600 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.535769433Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-11T21:50:59.600 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.536539922Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-11T21:50:59.600 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.537325811Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-11T21:50:59.600 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.538121195Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-11T21:50:59.600 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.538938509Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-11T21:50:59.600 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.539710028Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-11T21:50:59.600 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.54053316Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-11T21:50:59.600 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.541349932Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-11T21:50:59.600 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.542171299Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-11T21:50:59.600 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.542960369Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-11T21:50:59.600 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.543757337Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-11T21:50:59.600 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.544525853Z level=info msg="Executing migration" id="Add uid column" 2024-03-11T21:50:59.600 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.545326923Z level=info msg="Executing migration" id="Update uid value" 2024-03-11T21:50:59.600 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.54613737Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-11T21:50:59.601 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.546976699Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-11T21:50:59.601 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.547754249Z level=info msg="Executing migration" id="create api_key table" 2024-03-11T21:50:59.601 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.548534159Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-11T21:50:59.601 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.549319141Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-11T21:50:59.601 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.550103121Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-11T21:50:59.601 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.550888294Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-11T21:50:59.601 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.551699926Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-11T21:50:59.601 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.552487487Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-11T21:50:59.601 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.552913078Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-11T21:50:59.601 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.555400357Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-11T21:50:59.601 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.558028167Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-11T21:50:59.601 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.558493995Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-11T21:50:59.601 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.558915197Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-11T21:50:59.601 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.55936019Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-11T21:50:59.602 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.559670068Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-11T21:50:59.602 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.560011387Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-11T21:50:59.602 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.56032328Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-11T21:50:59.602 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.560839537Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-11T21:50:59.602 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.56142287Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-11T21:50:59.602 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.561754886Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-11T21:50:59.602 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.562324342Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-11T21:50:59.602 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.562860873Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-11T21:50:59.602 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.563251684Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-11T21:50:59.602 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.563610886Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-11T21:50:59.602 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.566252466Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-11T21:50:59.602 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.566671438Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-11T21:50:59.602 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.568794648Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-11T21:50:59.602 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.57120317Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-11T21:50:59.603 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.571485851Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-11T21:50:59.603 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.574076298Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-11T21:50:59.603 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.574619704Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-11T21:50:59.603 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.57674119Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-11T21:50:59.603 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.579281222Z level=info msg="Executing migration" id="create quota table v1" 2024-03-11T21:50:59.603 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.579663572Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-11T21:50:59.603 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.582273863Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-11T21:50:59.603 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.584662279Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-11T21:50:59.603 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.587128621Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-11T21:50:59.603 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.587558703Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-11T21:50:59.603 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.589937417Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-11T21:50:59.603 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.592410206Z level=info msg="Executing migration" id="create session table" 2024-03-11T21:50:59.603 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.59479523Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-11T21:50:59.603 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.595111174Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-11T21:50:59.604 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.597530427Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-11T21:50:59.604 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.600115863Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-11T21:50:59.604 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.600882516Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-11T21:50:59.604 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.601190513Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-11T21:50:59.736 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:59 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-c[27909]: 2024-03-11T21:50:59.488+0000 7f27069b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:50:59.737 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:59 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-c[27909]: 2024-03-11T21:50:59.488+0000 7f27069b9700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T21:50:59.843 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.601980506Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-11T21:50:59.843 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.602288314Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-11T21:50:59.843 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.602786418Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-11T21:50:59.843 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.603190218Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-11T21:50:59.843 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.603517271Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-11T21:50:59.843 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.604128635Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-11T21:50:59.844 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.60488384Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-11T21:50:59.844 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.605670809Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-11T21:50:59.844 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.607066668Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-11T21:50:59.844 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.60738805Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-11T21:50:59.844 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.608183793Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-11T21:50:59.844 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.608977706Z level=info msg="Executing migration" id="create alert table v1" 2024-03-11T21:50:59.844 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.60966706Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-11T21:50:59.844 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.610160673Z level=info msg="Executing migration" id="add index alert state" 2024-03-11T21:50:59.844 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.610919928Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-11T21:50:59.844 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.611703803Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-11T21:50:59.844 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.612472325Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-11T21:50:59.844 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.612912599Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-11T21:50:59.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.613685639Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-11T21:50:59.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.615053206Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-11T21:50:59.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.615828Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-11T21:50:59.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.616254898Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-11T21:50:59.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.617056574Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-11T21:50:59.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.617843472Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-11T21:50:59.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.618619471Z level=info msg="Executing migration" id="Add column is_default" 2024-03-11T21:50:59.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.61939898Z level=info msg="Executing migration" id="Add column frequency" 2024-03-11T21:50:59.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.620188532Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-11T21:50:59.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.620974908Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-11T21:50:59.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.621743251Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-11T21:50:59.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.622185977Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-11T21:50:59.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.622966372Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-11T21:50:59.846 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.623746238Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-11T21:50:59.846 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.624592676Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-11T21:50:59.846 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.625034613Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-11T21:50:59.846 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.627903857Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-11T21:50:59.846 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.630582597Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-11T21:50:59.846 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.633263526Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-11T21:50:59.846 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.633914741Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-11T21:50:59.846 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.636813195Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-11T21:50:59.846 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.639205039Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-11T21:50:59.846 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.641848255Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-11T21:50:59.846 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.64230702Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-11T21:50:59.847 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.645067643Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-11T21:50:59.847 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.647776796Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-11T21:50:59.847 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.650441989Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-11T21:50:59.847 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.650883703Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-11T21:50:59.847 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.653610376Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-11T21:50:59.847 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.656284671Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-11T21:50:59.847 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.65891165Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-11T21:50:59.847 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.659373922Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-11T21:50:59.847 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.662171004Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-11T21:50:59.847 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.664603756Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-11T21:50:59.847 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.667054077Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-11T21:50:59.847 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.667363017Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-11T21:50:59.847 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.669765343Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-11T21:50:59.847 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.670232496Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-11T21:50:59.848 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.67212465Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-11T21:50:59.849 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.675078762Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-11T21:50:59.849 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.675526392Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-11T21:50:59.849 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.678057546Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-11T21:50:59.849 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.679633081Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-11T21:50:59.850 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.680660386Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-11T21:50:59.850 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.683074956Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-11T21:50:59.850 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.683405199Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-11T21:50:59.850 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.685867655Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-11T21:50:59.850 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.686199646Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-11T21:50:59.850 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.688387863Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-11T21:50:59.850 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.690857802Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-11T21:50:59.850 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.691376468Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-11T21:50:59.850 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.69230217Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-11T21:50:59.850 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.692934573Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-11T21:50:59.850 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.693682096Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-11T21:50:59.850 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.694140648Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-11T21:50:59.851 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.69490564Z level=info msg="Executing migration" id="Move region to single row" 2024-03-11T21:50:59.851 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.695751365Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-11T21:50:59.851 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.696417936Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-11T21:50:59.851 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.69684356Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-11T21:50:59.851 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.697620345Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-11T21:50:59.851 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.698400894Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-11T21:50:59.851 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.699160719Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-11T21:50:59.851 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.699580249Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-11T21:50:59.851 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.700335896Z level=info msg="Executing migration" id="create test_data table" 2024-03-11T21:50:59.851 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.700800834Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-11T21:50:59.851 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.701198273Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-11T21:50:59.851 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.701981748Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-11T21:50:59.851 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.702395961Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-11T21:50:59.851 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.703177864Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-11T21:50:59.852 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.703501055Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-11T21:50:59.852 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.70399728Z level=info msg="Executing migration" id="create team table" 2024-03-11T21:50:59.852 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.704384827Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-11T21:50:59.852 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.704835502Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-11T21:50:59.852 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.705324123Z level=info msg="Executing migration" id="create team member table" 2024-03-11T21:50:59.852 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.706109294Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-11T21:50:59.852 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.706960941Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-11T21:50:59.852 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.707809813Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-11T21:50:59.852 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.708335067Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-11T21:50:59.852 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.709113559Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-11T21:50:59.852 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.709884493Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-11T21:50:59.852 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.71068983Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-11T21:50:59.852 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.711094377Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-11T21:50:59.853 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.711988271Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-11T21:50:59.853 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.712641923Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-11T21:50:59.853 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.71358593Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-11T21:50:59.853 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.714173307Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-11T21:50:59.853 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.717082076Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-11T21:50:59.853 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.719741399Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-11T21:50:59.853 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.722402019Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-11T21:50:59.853 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.725017085Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-11T21:50:59.853 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.727715693Z level=info msg="Executing migration" id="create tag table" 2024-03-11T21:50:59.853 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.730339918Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-11T21:50:59.853 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.730817676Z level=info msg="Executing migration" id="create login attempt table" 2024-03-11T21:50:59.853 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.733668685Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-11T21:50:59.853 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.736499223Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-11T21:50:59.854 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.739153709Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-11T21:50:59.854 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.741270805Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-11T21:50:59.854 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.743859845Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-11T21:50:59.854 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.746227116Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-11T21:50:59.854 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.748618543Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-11T21:50:59.854 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.749009417Z level=info msg="Executing migration" id="create user auth table" 2024-03-11T21:50:59.854 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.751506298Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-11T21:50:59.854 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.753876279Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-11T21:50:59.854 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.75625995Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-11T21:50:59.855 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.757203024Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-11T21:50:59.855 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.759665494Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-11T21:50:59.855 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.765156968Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-11T21:50:59.855 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.766008234Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-11T21:50:59.855 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.766456709Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-11T21:50:59.855 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.767180105Z level=info msg="Executing migration" id="create server_lock table" 2024-03-11T21:50:59.855 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.767567845Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-11T21:50:59.855 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.767994456Z level=info msg="Executing migration" id="create user auth token table" 2024-03-11T21:50:59.855 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.768427551Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-11T21:50:59.855 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.768829639Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-11T21:50:59.855 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.769247943Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-11T21:50:59.855 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.769661173Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-11T21:50:59.856 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.770410977Z level=info msg="Executing migration" id="create cache_data table" 2024-03-11T21:50:59.856 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.77080113Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-11T21:50:59.856 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.771223129Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-11T21:50:59.856 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.771630348Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-11T21:50:59.856 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.772035113Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-11T21:50:59.856 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.772319793Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-11T21:50:59.856 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.772585028Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-11T21:50:59.856 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.773012967Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-11T21:50:59.856 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.773420436Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-11T21:50:59.856 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.773825952Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-11T21:50:59.856 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.774117908Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-11T21:50:59.856 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.774517913Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-11T21:50:59.856 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.774887857Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-11T21:50:59.856 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.775336335Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-11T21:50:59.857 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.775771577Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-11T21:50:59.857 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.77658997Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-11T21:50:59.857 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.776998354Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-11T21:50:59.857 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.777306347Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-11T21:50:59.857 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.777694006Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-11T21:50:59.857 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.778135726Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-11T21:50:59.857 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.778563503Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-11T21:50:59.857 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.778866122Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-11T21:50:59.857 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.779280393Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-11T21:50:59.857 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.779660544Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-11T21:50:59.857 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.780124224Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-11T21:50:59.858 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.780562199Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-11T21:50:59.858 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.781430994Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-11T21:50:59.858 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.781855531Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-11T21:50:59.858 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.782280246Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-11T21:50:59.858 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.785741351Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-11T21:50:59.858 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.789051148Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-11T21:50:59.858 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.789531245Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-11T21:50:59.858 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.789939579Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-11T21:50:59.858 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.790757921Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-11T21:50:59.858 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.791173978Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-11T21:50:59.858 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.792711213Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-11T21:50:59.858 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.793155184Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-11T21:50:59.858 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.793594878Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-11T21:50:59.858 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.793869933Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-11T21:50:59.859 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.794724372Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-11T21:50:59.859 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.795584061Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-11T21:50:59.859 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.796464686Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-11T21:50:59.859 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.796884174Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-11T21:50:59.859 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.797365565Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-11T21:50:59.859 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.799782724Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-11T21:50:59.859 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.80063838Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-11T21:50:59.859 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.801087101Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-11T21:50:59.859 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.801886885Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-11T21:50:59.859 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.802696684Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-11T21:50:59.859 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.8030028Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-11T21:50:59.859 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.803443561Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-11T21:50:59.859 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.803875098Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-11T21:50:59.859 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.804321793Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-11T21:50:59.860 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.804599123Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-11T21:50:59.860 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.805483247Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-11T21:50:59.860 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.807217288Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-11T21:50:59.860 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.814742569Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-11T21:50:59.860 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.81560518Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-11T21:50:59.860 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.816492026Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-11T21:50:59.860 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.817258806Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-11T21:50:59.860 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.818093091Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-11T21:50:59.860 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.818957877Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-11T21:50:59.860 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.819723952Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-11T21:50:59.860 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.820632236Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-11T21:50:59.860 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.821412016Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-11T21:50:59.860 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.822317156Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-11T21:50:59.860 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.823058722Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-11T21:50:59.860 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.823860744Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-11T21:50:59.861 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.824726173Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-11T21:50:59.861 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.825122674Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-11T21:50:59.861 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.825960505Z level=info msg="Executing migration" id="create alert_image table" 2024-03-11T21:50:59.861 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.8268328Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-11T21:50:59.861 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.827487309Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-11T21:50:59.861 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.827764807Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-11T21:50:59.861 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.828176746Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-11T21:50:59.861 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.828607896Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T21:50:59.861 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.828689699Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T21:50:59.861 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.829350121Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-11T21:50:59.861 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.829739975Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-11T21:50:59.861 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.830233702Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-11T21:50:59.861 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.830357211Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-11T21:50:59.861 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.830474415Z level=info msg="alerts found to migrate" alerts=0 2024-03-11T21:50:59.862 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.830992893Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-11T21:50:59.862 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.831779397Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-11T21:50:59.862 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.832682082Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-11T21:50:59.862 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.833331826Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-11T21:50:59.862 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.833804563Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-11T21:50:59.862 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.834616134Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-11T21:50:59.862 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.835454898Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-11T21:50:59.862 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.835895291Z level=info msg="Executing migration" id="create data_keys table" 2024-03-11T21:50:59.862 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.836370184Z level=info msg="Executing migration" id="create secrets table" 2024-03-11T21:50:59.862 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.836822359Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-11T21:50:59.862 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.841014489Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-11T21:50:59.862 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.841930178Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-11T21:50:59.862 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.842729817Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-11T21:50:59.862 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.846896638Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-11T21:50:59.862 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.851099821Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-11T21:50:59.863 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.853532328Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-11T21:51:00.045 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:50:59 smithi165 podman[39982]: 2024-03-11 21:50:59.745122111 +0000 UTC m=+0.389196368 container died ad1ae1a889304dd9951babd2a1ca526eedb311953d67553183256e402829b7b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-11T21:51:00.093 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.85622988Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-11T21:51:00.093 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.856590496Z level=info msg="Executing migration" id="create permission table" 2024-03-11T21:51:00.094 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.859337024Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-11T21:51:00.094 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.862079405Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-11T21:51:00.094 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.864715163Z level=info msg="Executing migration" id="create role table" 2024-03-11T21:51:00.094 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.865230321Z level=info msg="Executing migration" id="add column display_name" 2024-03-11T21:51:00.094 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.868028464Z level=info msg="Executing migration" id="add column group_name" 2024-03-11T21:51:00.094 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.870629037Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-11T21:51:00.094 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.873304819Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-11T21:51:00.094 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.873753795Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-11T21:51:00.094 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.876343689Z level=info msg="Executing migration" id="create team role table" 2024-03-11T21:51:00.097 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.879041981Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-11T21:51:00.097 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.881778123Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-11T21:51:00.097 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.882260421Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-11T21:51:00.097 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.884973784Z level=info msg="Executing migration" id="create user role table" 2024-03-11T21:51:00.097 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.887776229Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-11T21:51:00.097 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.890186076Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-11T21:51:00.097 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.890655654Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-11T21:51:00.097 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.893137203Z level=info msg="Executing migration" id="create builtin role table" 2024-03-11T21:51:00.098 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.895613217Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-11T21:51:00.098 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.898020303Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-11T21:51:00.098 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.89854926Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-11T21:51:00.098 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.901067246Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-11T21:51:00.098 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.901538487Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-11T21:51:00.098 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.901984141Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-11T21:51:00.098 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.90241273Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-11T21:51:00.098 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.905167216Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-11T21:51:00.098 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.90556397Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-11T21:51:00.098 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.907903497Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-11T21:51:00.098 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.910579336Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-11T21:51:00.098 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.911006547Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-11T21:51:00.099 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.913798908Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-11T21:51:00.099 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.916201092Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-11T21:51:00.099 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.918641543Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-11T21:51:00.099 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.919116673Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-11T21:51:00.099 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.921655547Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-11T21:51:00.099 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.924054633Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-11T21:51:00.099 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.926421564Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-11T21:51:00.100 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.926838883Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-11T21:51:00.100 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.929380394Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-11T21:51:00.100 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.9318795Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-11T21:51:00.100 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.934423108Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-11T21:51:00.100 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.934849936Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-11T21:51:00.100 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.937317142Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-11T21:51:00.100 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.93979473Z level=info msg="Executing migration" id="add correlation config column" 2024-03-11T21:51:00.100 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.942166805Z level=info msg="Executing migration" id="create entity_events table" 2024-03-11T21:51:00.100 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.942627625Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-11T21:51:00.100 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.945132543Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T21:51:00.101 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.945228672Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T21:51:00.101 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.947714776Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T21:51:00.101 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.947806378Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T21:51:00.101 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.948480875Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-11T21:51:00.101 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.948875949Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-11T21:51:00.101 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.949648281Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-11T21:51:00.101 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.950450874Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T21:51:00.101 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.951229539Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-11T21:51:00.101 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.951668408Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T21:51:00.101 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.952367444Z level=info msg="Executing migration" id="Drop public config table" 2024-03-11T21:51:00.101 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.953149543Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-11T21:51:00.101 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.953928389Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-11T21:51:00.101 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.954380099Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T21:51:00.101 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.955122541Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-11T21:51:00.101 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.955570603Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-11T21:51:00.102 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.958520307Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-11T21:51:00.102 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.959622084Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-11T21:51:00.102 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.960884663Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-11T21:51:00.102 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.961626147Z level=info msg="Executing migration" id="add share column" 2024-03-11T21:51:00.102 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.962618618Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-11T21:51:00.102 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.963404571Z level=info msg="Executing migration" id="create file table" 2024-03-11T21:51:00.102 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.963835686Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-11T21:51:00.102 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.964639102Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-11T21:51:00.102 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.965445454Z level=info msg="Executing migration" id="create file_meta table" 2024-03-11T21:51:00.102 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.966258754Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-11T21:51:00.102 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.966696435Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-11T21:51:00.102 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.967440991Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-11T21:51:00.102 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.96780872Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-11T21:51:00.102 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.968242443Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-11T21:51:00.103 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.969029286Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-11T21:51:00.103 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.97009333Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-11T21:51:00.103 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.970870339Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-11T21:51:00.103 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.971643792Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-11T21:51:00.103 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.97245649Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-11T21:51:00.103 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.972783586Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-11T21:51:00.103 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.97356404Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-11T21:51:00.103 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.974590326Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-11T21:51:00.103 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.975591709Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-11T21:51:00.103 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.976017742Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-11T21:51:00.103 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.986078743Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-11T21:51:00.103 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.988767035Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-11T21:51:00.103 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.991472222Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-11T21:51:00.103 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.994865786Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-11T21:51:00.104 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:50:59 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:50:59.997491214Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-11T21:51:00.104 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:51:00.000116666Z level=info msg="Executing migration" id="create folder table" 2024-03-11T21:51:00.104 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:51:00.002738774Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-11T21:51:00.104 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:51:00.003219145Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-11T21:51:00.104 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:51:00.006026057Z level=info msg="Executing migration" id="Update folder title length" 2024-03-11T21:51:00.104 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:51:00.008808082Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-11T21:51:00.104 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=migrator t=2024-03-11T21:51:00.011470332Z level=info msg="migrations completed" performed=485 skipped=0 duration=606.378159ms 2024-03-11T21:51:00.104 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=sqlstore t=2024-03-11T21:51:00.011900186Z level=info msg="Created default organization" 2024-03-11T21:51:00.104 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=plugin.loader t=2024-03-11T21:51:00.07106893Z level=info msg="Plugin registered" pluginID=input 2024-03-11T21:51:00.104 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=plugin.loader t=2024-03-11T21:51:00.099327005Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-11T21:51:00.104 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=plugin.loader t=2024-03-11T21:51:00.099371623Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-11T21:51:00.104 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=secrets t=2024-03-11T21:51:00.099409046Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-11T21:51:00.104 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=query_data t=2024-03-11T21:51:00.101591405Z level=info msg="Query Service initialization" 2024-03-11T21:51:00.321 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:51:00 smithi165 podman[39982]: 2024-03-11 21:51:00.144973095 +0000 UTC m=+0.789047360 container cleanup ad1ae1a889304dd9951babd2a1ca526eedb311953d67553183256e402829b7b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-c, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T21:51:00.321 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 21:51:00 smithi165 bash[39982]: ceph-85326acc-dff0-11ee-95c8-87774f69a715-mon-c 2024-03-11T21:51:00.424 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=live.push_http t=2024-03-11T21:51:00.106647322Z level=info msg="Live Push Gateway initialization" 2024-03-11T21:51:00.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T21:51:00.503 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:00.504 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-11T21:51:00.504 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T21:51:00.504 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-85326acc-dff0-11ee-95c8-87774f69a715@mgr.y 2024-03-11T21:51:00.869 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:51:00 smithi002 systemd[1]: Stopping Ceph mgr.y for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:51:01.136 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:51:00 smithi002 podman[52141]: 2024-03-11 21:51:00.881735118 +0000 UTC m=+0.145211395 container died 0aba717a9141167a8cc57fc640ddbb6abca20f823152e3eb099e9acced4aa2d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-11T21:51:01.148 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=infra.usagestats.collector t=2024-03-11T21:51:00.748298683Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-11T21:51:01.148 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=provisioning.datasources t=2024-03-11T21:51:00.749390896Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-11T21:51:01.148 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=provisioning.datasources t=2024-03-11T21:51:00.760286181Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-11T21:51:01.148 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=provisioning.alerting t=2024-03-11T21:51:00.771185339Z level=info msg="starting to provision alerting" 2024-03-11T21:51:01.149 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=provisioning.alerting t=2024-03-11T21:51:00.771204937Z level=info msg="finished to provision alerting" 2024-03-11T21:51:01.149 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=grafanaStorageLogger t=2024-03-11T21:51:00.771371742Z level=info msg="storage starting" 2024-03-11T21:51:01.149 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=ngalert.state.manager t=2024-03-11T21:51:00.771470641Z level=info msg="Warming state cache for startup" 2024-03-11T21:51:01.149 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=ngalert.state.manager t=2024-03-11T21:51:00.771575307Z level=info msg="State cache has been initialized" states=0 duration=103.7µs 2024-03-11T21:51:01.149 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=ticker t=2024-03-11T21:51:00.771631019Z level=info msg=starting first_tick=2024-03-11T21:51:10Z 2024-03-11T21:51:01.149 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=ngalert.multiorg.alertmanager t=2024-03-11T21:51:00.771655996Z level=info msg="starting MultiOrg Alertmanager" 2024-03-11T21:51:01.149 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=http.server t=2024-03-11T21:51:00.773192732Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-11T21:51:01.149 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=sqlstore.transactions t=2024-03-11T21:51:00.79238671Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T21:51:01.149 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=sqlstore.transactions t=2024-03-11T21:51:00.805932525Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-11T21:51:01.149 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=sqlstore.transactions t=2024-03-11T21:51:00.817065443Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-11T21:51:01.149 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=sqlstore.transactions t=2024-03-11T21:51:00.827421022Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-03-11T21:51:01.149 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:00 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a[38984]: logger=sqlstore.transactions t=2024-03-11T21:51:00.837931231Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4 code="database is locked" 2024-03-11T21:51:01.395 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:51:01 smithi002 podman[52141]: 2024-03-11 21:51:01.255361806 +0000 UTC m=+0.518838085 container cleanup 0aba717a9141167a8cc57fc640ddbb6abca20f823152e3eb099e9acced4aa2d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-11T21:51:01.396 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:51:01 smithi002 bash[52141]: ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y 2024-03-11T21:51:02.145 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Mar 11 21:51:02 smithi002 podman[52170]: 2024-03-11 21:51:02.059939416 +0000 UTC m=+1.177237473 container remove 0aba717a9141167a8cc57fc640ddbb6abca20f823152e3eb099e9acced4aa2d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-y, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-11T21:51:02.198 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T21:51:02.237 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:02.237 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T21:51:02.237 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T21:51:02.237 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-85326acc-dff0-11ee-95c8-87774f69a715@mgr.x 2024-03-11T21:51:02.558 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:51:02 smithi122 systemd[1]: Stopping Ceph mgr.x for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:51:02.810 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:51:02 smithi122 podman[39211]: 2024-03-11 21:51:02.566470559 +0000 UTC m=+0.130106904 container died 31ff6547d310e16cd8ab69f9757d3dbd86cf5453e600cd8222ba885d6266bb35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-11T21:51:03.077 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:51:02 smithi122 podman[39211]: 2024-03-11 21:51:02.900735864 +0000 UTC m=+0.464372208 container cleanup 31ff6547d310e16cd8ab69f9757d3dbd86cf5453e600cd8222ba885d6266bb35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-11T21:51:03.077 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Mar 11 21:51:02 smithi122 bash[39211]: ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-x 2024-03-11T21:51:03.299 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T21:51:03.337 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:03.337 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T21:51:03.337 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-11T21:51:03.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-85326acc-dff0-11ee-95c8-87774f69a715@mgr.z 2024-03-11T21:51:03.673 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:51:03 smithi165 systemd[1]: Stopping Ceph mgr.z for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:51:03.965 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:51:03 smithi165 podman[40111]: 2024-03-11 21:51:03.680607404 +0000 UTC m=+0.145001716 container died 528f350cb9e22c624add3dcebfc8c12f0fea2bf1303f50b890bf0dce0c5dc84d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-11T21:51:04.290 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:51:04 smithi165 podman[40111]: 2024-03-11 21:51:04.056149255 +0000 UTC m=+0.520543574 container cleanup 528f350cb9e22c624add3dcebfc8c12f0fea2bf1303f50b890bf0dce0c5dc84d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-11T21:51:04.290 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 21:51:04 smithi165 bash[40111]: ceph-85326acc-dff0-11ee-95c8-87774f69a715-mgr-z 2024-03-11T21:51:04.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@mgr.z.service' 2024-03-11T21:51:04.450 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:04.451 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-11T21:51:04.451 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T21:51:04.451 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-85326acc-dff0-11ee-95c8-87774f69a715@osd.0 2024-03-11T21:51:04.749 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:51:04 smithi002 systemd[1]: Stopping Ceph osd.0 for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:51:05.146 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:51:04 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-0[38233]: 2024-03-11T21:51:04.756+0000 7fb884096700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:51:05.146 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:51:04 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-0[38233]: 2024-03-11T21:51:04.756+0000 7fb884096700 -1 osd.0 31 *** Got signal Terminated *** 2024-03-11T21:51:05.146 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:51:04 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-0[38233]: 2024-03-11T21:51:04.756+0000 7fb884096700 -1 osd.0 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:51:10.646 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:51:10 smithi002 podman[52350]: 2024-03-11 21:51:10.25504288 +0000 UTC m=+5.608659917 container died 074468c85393fcbe9cb580cbeb6a035da4b13185cb146952bf94c2a04da2a586 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-11T21:51:10.910 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:51:10 smithi002 podman[52350]: 2024-03-11 21:51:10.6598923 +0000 UTC m=+6.013509383 container cleanup 074468c85393fcbe9cb580cbeb6a035da4b13185cb146952bf94c2a04da2a586 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-11T21:51:10.911 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:51:10 smithi002 bash[52350]: ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-0 2024-03-11T21:51:10.911 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:51:10 smithi002 podman[52770]: 2024-03-11 21:51:10.869422012 +0000 UTC m=+0.611772156 container remove 074468c85393fcbe9cb580cbeb6a035da4b13185cb146952bf94c2a04da2a586 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-03-11T21:51:11.489 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:51:11 smithi002 podman[52872]: 2024-03-11 21:51:11.088104171 +0000 UTC m=+0.018244634 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:51:11.489 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:51:11 smithi002 podman[52872]: 2024-03-11 21:51:11.203394814 +0000 UTC m=+0.133535258 container create 84ad7b1f1141aec846d859ceb50e2e3cb6b50fef6e330d8c0da396bb3ce4a007 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5) 2024-03-11T21:51:11.489 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:51:11 smithi002 podman[52872]: 2024-03-11 21:51:11.414062071 +0000 UTC m=+0.344202518 container init 84ad7b1f1141aec846d859ceb50e2e3cb6b50fef6e330d8c0da396bb3ce4a007 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-11T21:51:11.489 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:51:11 smithi002 podman[52872]: 2024-03-11 21:51:11.419574216 +0000 UTC m=+0.349714663 container start 84ad7b1f1141aec846d859ceb50e2e3cb6b50fef6e330d8c0da396bb3ce4a007 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T21:51:11.489 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:51:11 smithi002 podman[52872]: 2024-03-11 21:51:11.497198363 +0000 UTC m=+0.427338816 container attach 84ad7b1f1141aec846d859ceb50e2e3cb6b50fef6e330d8c0da396bb3ce4a007 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-11T21:51:11.895 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 21:51:11 smithi002 podman[52872]: 2024-03-11 21:51:11.709316825 +0000 UTC m=+0.639457264 container died 84ad7b1f1141aec846d859ceb50e2e3cb6b50fef6e330d8c0da396bb3ce4a007 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-11T21:51:12.457 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T21:51:12.501 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:12.501 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T21:51:12.501 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T21:51:12.502 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-85326acc-dff0-11ee-95c8-87774f69a715@osd.1 2024-03-11T21:51:12.674 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:51:12 smithi122 systemd[1]: Stopping Ceph osd.1 for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:51:13.674 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:51:13 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-1[33104]: 2024-03-11T21:51:13.617+0000 7f8042bc0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:51:13.675 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:51:13 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-1[33104]: 2024-03-11T21:51:13.617+0000 7f8042bc0700 -1 osd.1 31 *** Got signal Terminated *** 2024-03-11T21:51:13.675 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:51:13 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-1[33104]: 2024-03-11T21:51:13.617+0000 7f8042bc0700 -1 osd.1 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:51:19.344 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:51:19 smithi122 podman[39773]: 2024-03-11 21:51:19.101263728 +0000 UTC m=+5.721827475 container died ecdbfc322e36ff31642f8281b51d69bf5840b7b63bec76b3f4594c15a53f8220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-1, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:51:19.674 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:51:19 smithi122 podman[39773]: 2024-03-11 21:51:19.458187416 +0000 UTC m=+6.078751162 container cleanup ecdbfc322e36ff31642f8281b51d69bf5840b7b63bec76b3f4594c15a53f8220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-11T21:51:19.675 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:51:19 smithi122 bash[39773]: ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-1 2024-03-11T21:51:20.060 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:51:19 smithi122 podman[40021]: 2024-03-11 21:51:19.726222599 +0000 UTC m=+0.627149561 container remove ecdbfc322e36ff31642f8281b51d69bf5840b7b63bec76b3f4594c15a53f8220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397) 2024-03-11T21:51:20.329 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:51:20 smithi122 podman[40117]: 2024-03-11 21:51:20.068697462 +0000 UTC m=+0.109125876 container create b5a9e423de6d1718b663076acb1f72ca8d575f174873efa02a887d886de5e2a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_BRANCH=HEAD) 2024-03-11T21:51:20.329 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:51:20 smithi122 podman[40117]: 2024-03-11 21:51:19.978696643 +0000 UTC m=+0.019125071 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:51:20.329 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:51:20 smithi122 podman[40117]: 2024-03-11 21:51:20.337510513 +0000 UTC m=+0.377938939 container init b5a9e423de6d1718b663076acb1f72ca8d575f174873efa02a887d886de5e2a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:51:20.618 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:51:20 smithi122 podman[40117]: 2024-03-11 21:51:20.34338166 +0000 UTC m=+0.383810076 container start b5a9e423de6d1718b663076acb1f72ca8d575f174873efa02a887d886de5e2a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-1-deactivate, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-11T21:51:20.619 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:51:20 smithi122 podman[40117]: 2024-03-11 21:51:20.429227049 +0000 UTC m=+0.469655474 container attach b5a9e423de6d1718b663076acb1f72ca8d575f174873efa02a887d886de5e2a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-11T21:51:20.907 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:51:20 smithi122 podman[40117]: 2024-03-11 21:51:20.627531737 +0000 UTC m=+0.667960157 container died b5a9e423de6d1718b663076acb1f72ca8d575f174873efa02a887d886de5e2a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-11T21:51:21.165 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 11 21:51:21 smithi122 podman[40163]: 2024-03-11 21:51:21.157071003 +0000 UTC m=+0.521410467 container remove b5a9e423de6d1718b663076acb1f72ca8d575f174873efa02a887d886de5e2a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-11T21:51:21.169 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T21:51:21.209 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:21.209 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T21:51:21.210 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T21:51:21.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-85326acc-dff0-11ee-95c8-87774f69a715@osd.2 2024-03-11T21:51:21.509 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:21 smithi165 systemd[1]: Stopping Ceph osd.2 for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:51:21.833 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:21 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-2[34500]: 2024-03-11T21:51:21.514+0000 7fb2a04b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:51:21.833 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:21 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-2[34500]: 2024-03-11T21:51:21.514+0000 7fb2a04b1700 -1 osd.2 31 *** Got signal Terminated *** 2024-03-11T21:51:21.833 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:21 smithi165 ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-2[34500]: 2024-03-11T21:51:21.514+0000 7fb2a04b1700 -1 osd.2 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:51:27.083 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:26 smithi165 podman[40715]: 2024-03-11 21:51:26.82710922 +0000 UTC m=+5.415299893 container died ed0cc7a46dff5ba1e20d7ae8228c002303d00215ef0a8703f5cd27cb77c2ed47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, ceph=True) 2024-03-11T21:51:27.371 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:27 smithi165 podman[40715]: 2024-03-11 21:51:27.162098953 +0000 UTC m=+5.750289626 container cleanup ed0cc7a46dff5ba1e20d7ae8228c002303d00215ef0a8703f5cd27cb77c2ed47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-2, RELEASE=squid-94a5397, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-11T21:51:27.371 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:27 smithi165 bash[40715]: ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-2 2024-03-11T21:51:27.371 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:27 smithi165 podman[40737]: 2024-03-11 21:51:27.312643399 +0000 UTC m=+0.482869443 container remove ed0cc7a46dff5ba1e20d7ae8228c002303d00215ef0a8703f5cd27cb77c2ed47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-11T21:51:27.982 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:27 smithi165 podman[40942]: 2024-03-11 21:51:27.738363491 +0000 UTC m=+0.086471522 container create 33588bbcae5814c653bf49701dc71ac2ad35d6b5c7b666bf2b9851095689b978 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:51:27.982 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:27 smithi165 podman[40942]: 2024-03-11 21:51:27.67082784 +0000 UTC m=+0.018935863 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T21:51:27.982 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:27 smithi165 podman[40942]: 2024-03-11 21:51:27.924941345 +0000 UTC m=+0.273049366 container init 33588bbcae5814c653bf49701dc71ac2ad35d6b5c7b666bf2b9851095689b978 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-2-deactivate, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-11T21:51:27.982 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:27 smithi165 podman[40942]: 2024-03-11 21:51:27.93085807 +0000 UTC m=+0.278966088 container start 33588bbcae5814c653bf49701dc71ac2ad35d6b5c7b666bf2b9851095689b978 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T21:51:27.982 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:27 smithi165 podman[40942]: 2024-03-11 21:51:27.990514996 +0000 UTC m=+0.338623023 container attach 33588bbcae5814c653bf49701dc71ac2ad35d6b5c7b666bf2b9851095689b978 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:51:28.526 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:28 smithi165 podman[40942]: 2024-03-11 21:51:28.252301561 +0000 UTC m=+0.600409580 container died 33588bbcae5814c653bf49701dc71ac2ad35d6b5c7b666bf2b9851095689b978 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:51:28.771 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T21:51:28.799 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:28 smithi165 podman[41046]: 2024-03-11 21:51:28.751402966 +0000 UTC m=+0.493583974 container remove 33588bbcae5814c653bf49701dc71ac2ad35d6b5c7b666bf2b9851095689b978 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:51:28.799 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:28 smithi165 systemd[1]: ceph-85326acc-dff0-11ee-95c8-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-11T21:51:28.799 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:28 smithi165 systemd[1]: Stopped Ceph osd.2 for 85326acc-dff0-11ee-95c8-87774f69a715. 2024-03-11T21:51:28.799 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 21:51:28 smithi165 systemd[1]: ceph-85326acc-dff0-11ee-95c8-87774f69a715@osd.2.service: Consumed 3.846s CPU time. 2024-03-11T21:51:28.815 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:28.816 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T21:51:28.816 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-11T21:51:28.816 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-85326acc-dff0-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-11T21:51:29.097 INFO:journalctl@ceph.rgw.foo.a.smithi002.stdout:Mar 11 21:51:28 smithi002 systemd[1]: Stopping Ceph rgw.foo.a for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:51:29.098 INFO:journalctl@ceph.rgw.foo.a.smithi002.stdout:Mar 11 21:51:29 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-rgw-foo-a[41146]: 2024-03-11T21:51:29.104+0000 7f3b217bd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:51:29.098 INFO:journalctl@ceph.rgw.foo.a.smithi002.stdout:Mar 11 21:51:29 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-rgw-foo-a[41146]: 2024-03-11T21:51:29.104+0000 7f3b3002dac0 -1 shutting down 2024-03-11T21:51:39.146 INFO:journalctl@ceph.rgw.foo.a.smithi002.stdout:Mar 11 21:51:39 smithi002 bash[53580]: time="2024-03-11T21:51:39Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-85326acc-dff0-11ee-95c8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-11T21:51:39.146 INFO:journalctl@ceph.rgw.foo.a.smithi002.stdout:Mar 11 21:51:39 smithi002 podman[53580]: 2024-03-11 21:51:39.132993935 +0000 UTC m=+10.116439873 container died 9837194d9807bd57d8bb0a9bcb619d47f294204e7d1b5bea2224bccd7a0f944f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T21:51:39.748 INFO:journalctl@ceph.rgw.foo.a.smithi002.stdout:Mar 11 21:51:39 smithi002 podman[53580]: 2024-03-11 21:51:39.596776052 +0000 UTC m=+10.580222030 container cleanup 9837194d9807bd57d8bb0a9bcb619d47f294204e7d1b5bea2224bccd7a0f944f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-rgw-foo-a, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-11T21:51:39.749 INFO:journalctl@ceph.rgw.foo.a.smithi002.stdout:Mar 11 21:51:39 smithi002 bash[53580]: ceph-85326acc-dff0-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-11T21:51:40.645 INFO:journalctl@ceph.rgw.foo.a.smithi002.stdout:Mar 11 21:51:40 smithi002 bash[53728]: ceph-85326acc-dff0-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-11T21:51:40.754 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-11T21:51:40.795 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:40.795 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-11T21:51:40.795 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-11T21:51:40.795 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-85326acc-dff0-11ee-95c8-87774f69a715@prometheus.a 2024-03-11T21:51:41.079 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:40 smithi122 systemd[1]: Stopping Ceph prometheus.a for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:51:41.360 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:41 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:51:41.087Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:51:41.360 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:41 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:51:41.087Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T21:51:41.360 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:41 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:51:41.087Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T21:51:41.360 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:41 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:51:41.087Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T21:51:41.361 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:41 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:51:41.087Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T21:51:41.361 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:41 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:51:41.087Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T21:51:41.361 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:41 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:51:41.087Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T21:51:41.361 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:41 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:51:41.087Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T21:51:41.361 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:41 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:51:41.087Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T21:51:41.361 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:41 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:51:41.088Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T21:51:41.361 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:41 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:51:41.088Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T21:51:41.361 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:41 smithi122 ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a[35949]: ts=2024-03-11T21:51:41.088Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T21:51:41.361 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:41 smithi122 podman[40872]: 2024-03-11 21:51:41.098090277 +0000 UTC m=+0.098827626 container died 0becd75e25cc959b368ea5ce756cab8701c16b6d00d2a11f30d641f79f58220c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:51:41.674 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:41 smithi122 podman[40872]: 2024-03-11 21:51:41.467687125 +0000 UTC m=+0.468424482 container cleanup 0becd75e25cc959b368ea5ce756cab8701c16b6d00d2a11f30d641f79f58220c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:51:41.675 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Mar 11 21:51:41 smithi122 bash[40872]: ceph-85326acc-dff0-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T21:51:41.827 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-85326acc-dff0-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-11T21:51:41.865 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:41.865 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-11T21:51:41.865 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 85326acc-dff0-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:51:42.049 INFO:teuthology.orchestra.run.smithi002.stdout:Deleting cluster with fsid: 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:51:43.396 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:51:43 smithi002 systemd[1]: Stopping Ceph alertmanager.a for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:51:43.396 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:51:43 smithi002 ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a[49127]: ts=2024-03-11T21:51:43.304Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:51:43.396 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:51:43 smithi002 podman[54170]: 2024-03-11 21:51:43.325448283 +0000 UTC m=+0.098976191 container died d16e62793684e11801b20d27246bcae26defa639761a068ee3fc886765c44a89 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:51:43.896 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:51:43 smithi002 podman[54170]: 2024-03-11 21:51:43.744097143 +0000 UTC m=+0.517625051 container cleanup d16e62793684e11801b20d27246bcae26defa639761a068ee3fc886765c44a89 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:51:43.896 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:51:43 smithi002 bash[54170]: ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T21:51:44.196 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:51:43 smithi002 podman[54189]: 2024-03-11 21:51:43.918386616 +0000 UTC m=+0.592176447 container remove d16e62793684e11801b20d27246bcae26defa639761a068ee3fc886765c44a89 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:51:44.196 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:51:43 smithi002 podman[54189]: 2024-03-11 21:51:43.979603645 +0000 UTC m=+0.653393487 volume remove a552620b1b899cc5768a0ff887f6787280b32ed43b95ba82bc894a99351c0990 2024-03-11T21:51:44.476 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:51:44 smithi002 systemd[1]: ceph-85326acc-dff0-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T21:51:44.477 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:51:44 smithi002 systemd[1]: Stopped Ceph alertmanager.a for 85326acc-dff0-11ee-95c8-87774f69a715. 2024-03-11T21:51:44.477 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Mar 11 21:51:44 smithi002 systemd[1]: ceph-85326acc-dff0-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 1.027s CPU time. 2024-03-11T21:51:44.887 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 85326acc-dff0-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:51:45.074 INFO:teuthology.orchestra.run.smithi122.stdout:Deleting cluster with fsid: 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:51:46.370 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:46 smithi122 systemd[1]: Stopping Ceph grafana.a for 85326acc-dff0-11ee-95c8-87774f69a715... 2024-03-11T21:51:46.626 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:46 smithi122 podman[41253]: 2024-03-11 21:51:46.377503787 +0000 UTC m=+0.116866333 container died c7386cf0e58091466c1cef980c93b998cbff1266bc82df0f566d4f572bd6937c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, version=8.5, maintainer=Paul Cuzner , release=239.1651231664, build-date=2022-04-29T11:48:11.055897, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, description=Ceph Grafana Container) 2024-03-11T21:51:46.925 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:46 smithi122 podman[41253]: 2024-03-11 21:51:46.696475421 +0000 UTC m=+0.435838013 container cleanup c7386cf0e58091466c1cef980c93b998cbff1266bc82df0f566d4f572bd6937c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a, com.redhat.component=ubi8-container, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, architecture=x86_64, vendor=Red Hat, Inc., vcs-type=git, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, distribution-scope=public, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-11T21:51:46.925 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:46 smithi122 bash[41253]: ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a 2024-03-11T21:51:46.925 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:46 smithi122 podman[41272]: 2024-03-11 21:51:46.872299482 +0000 UTC m=+0.494217996 container remove c7386cf0e58091466c1cef980c93b998cbff1266bc82df0f566d4f572bd6937c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-85326acc-dff0-11ee-95c8-87774f69a715-grafana-a, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., io.openshift.expose-services=, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, vcs-type=git, io.buildah.version=1.32.0, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, release=239.1651231664, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-11T21:51:46.925 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:46 smithi122 systemd[1]: ceph-85326acc-dff0-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:51:47.424 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:47 smithi122 systemd[1]: ceph-85326acc-dff0-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-11T21:51:47.425 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:47 smithi122 systemd[1]: Stopped Ceph grafana.a for 85326acc-dff0-11ee-95c8-87774f69a715. 2024-03-11T21:51:47.425 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Mar 11 21:51:47 smithi122 systemd[1]: ceph-85326acc-dff0-11ee-95c8-87774f69a715@grafana.a.service: Consumed 3.109s CPU time. 2024-03-11T21:51:47.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 85326acc-dff0-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:51:47.945 INFO:teuthology.orchestra.run.smithi165.stdout:Deleting cluster with fsid: 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:51:49.250 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:51:49.285 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:51:49.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:51:49.349 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T21:51:49.350 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928/remote/smithi002/crash 2024-03-11T21:51:49.351 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:51:49.395 INFO:teuthology.orchestra.run.smithi002.stderr:tar: /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:51:49.395 INFO:teuthology.orchestra.run.smithi002.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:51:49.397 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928/remote/smithi122/crash 2024-03-11T21:51:49.398 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:51:49.439 INFO:teuthology.orchestra.run.smithi122.stderr:tar: /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:51:49.440 INFO:teuthology.orchestra.run.smithi122.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:51:49.441 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928/remote/smithi165/crash 2024-03-11T21:51:49.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:51:49.485 INFO:teuthology.orchestra.run.smithi165.stderr:tar: /var/lib/ceph/85326acc-dff0-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:51:49.485 INFO:teuthology.orchestra.run.smithi165.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:51:49.486 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T21:51:49.490 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T21:51:49.522 INFO:teuthology.orchestra.run.smithi002.stdout:2024-03-11T21:47:43.460833+0000 mon.a (mon.0) 248 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T21:51:49.522 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T21:51:49.523 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[SEC\]' /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T21:51:49.593 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[ERR\]' /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T21:51:49.663 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[WRN\]' /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T21:51:49.732 INFO:teuthology.orchestra.run.smithi002.stdout:2024-03-11T21:47:43.460833+0000 mon.a (mon.0) 248 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T21:51:49.732 INFO:tasks.cephadm:Compressing logs... 2024-03-11T21:51:49.733 DEBUG:teuthology.orchestra.run.smithi002:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:51:49.776 DEBUG:teuthology.orchestra.run.smithi122:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:51:49.779 DEBUG:teuthology.orchestra.run.smithi165:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:51:49.801 INFO:teuthology.orchestra.run.smithi002.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:51:49.802 INFO:teuthology.orchestra.run.smithi002.stderr:: No such file or directory 2024-03-11T21:51:49.802 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T21:51:49.802 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:51:49.802 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-11T21:51:49.802 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T21:51:49.802 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T21:51:49.802 INFO:teuthology.orchestra.run.smithi002.stderr: 57.7% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-11T21:51:49.802 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:51:49.802 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.log 2024-03-11T21:51:49.802 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 89.7% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:51:49.802 INFO:teuthology.orchestra.run.smithi002.stderr: --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T21:51:49.802 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.log: 82.7% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:51:49.803 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-volume.log: 85.1% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:51:49.803 INFO:teuthology.orchestra.run.smithi002.stderr: 83.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:51:49.808 INFO:teuthology.orchestra.run.smithi165.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:51:49.808 INFO:teuthology.orchestra.run.smithi165.stderr:: No such file or directory 2024-03-11T21:51:49.808 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:51:49.808 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mgr.z.log 2024-03-11T21:51:49.808 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:51:49.809 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.log 2024-03-11T21:51:49.809 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T21:51:49.809 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.cephadm.log: /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T21:51:49.809 INFO:teuthology.orchestra.run.smithi165.stderr: 81.1% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:51:49.809 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-osd.2.log: 89.8% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:51:49.809 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T21:51:49.809 INFO:teuthology.orchestra.run.smithi165.stderr: 84.6% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:51:49.809 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-volume.log: 77.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:51:49.810 INFO:teuthology.orchestra.run.smithi122.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:51:49.810 INFO:teuthology.orchestra.run.smithi122.stderr:: No such file or directory 2024-03-11T21:51:49.811 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:51:49.811 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T21:51:49.811 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T21:51:49.811 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.audit.log: /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:51:49.812 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.log 2024-03-11T21:51:49.812 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T21:51:49.812 INFO:teuthology.orchestra.run.smithi122.stderr: 81.0% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:51:49.813 INFO:teuthology.orchestra.run.smithi122.stderr: 89.7% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:51:49.813 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T21:51:49.813 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-volume.log: 84.4% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:51:49.813 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mon.b.log: 76.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:51:49.813 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mon.c.log: 90.2% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mgr.z.log.gz 2024-03-11T21:51:49.815 INFO:teuthology.orchestra.run.smithi122.stderr: 90.3% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T21:51:49.818 INFO:teuthology.orchestra.run.smithi002.stderr: 90.2% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:51:49.823 INFO:teuthology.orchestra.run.smithi165.stderr: 90.2% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:51:49.826 INFO:teuthology.orchestra.run.smithi122.stderr: 90.2% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:51:49.872 INFO:teuthology.orchestra.run.smithi165.stderr: 92.4% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T21:51:49.881 INFO:teuthology.orchestra.run.smithi122.stderr: 92.4% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T21:51:49.882 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T21:51:50.008 INFO:teuthology.orchestra.run.smithi002.stderr: 91.5% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T21:51:50.069 INFO:teuthology.orchestra.run.smithi002.stderr: 94.3% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T21:51:50.071 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-03-11T21:51:50.071 INFO:teuthology.orchestra.run.smithi002.stderr:real 0m0.284s 2024-03-11T21:51:50.071 INFO:teuthology.orchestra.run.smithi002.stderr:user 0m0.574s 2024-03-11T21:51:50.071 INFO:teuthology.orchestra.run.smithi002.stderr:sys 0m0.040s 2024-03-11T21:51:50.075 INFO:teuthology.orchestra.run.smithi165.stderr: 94.3% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T21:51:50.077 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-03-11T21:51:50.077 INFO:teuthology.orchestra.run.smithi165.stderr:real 0m0.283s 2024-03-11T21:51:50.077 INFO:teuthology.orchestra.run.smithi165.stderr:user 0m0.355s 2024-03-11T21:51:50.077 INFO:teuthology.orchestra.run.smithi165.stderr:sys 0m0.032s 2024-03-11T21:51:50.125 INFO:teuthology.orchestra.run.smithi122.stderr: 94.3% -- replaced with /var/log/ceph/85326acc-dff0-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T21:51:50.127 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-11T21:51:50.127 INFO:teuthology.orchestra.run.smithi122.stderr:real 0m0.333s 2024-03-11T21:51:50.127 INFO:teuthology.orchestra.run.smithi122.stderr:user 0m0.405s 2024-03-11T21:51:50.127 INFO:teuthology.orchestra.run.smithi122.stderr:sys 0m0.042s 2024-03-11T21:51:50.127 INFO:tasks.cephadm:Archiving logs... 2024-03-11T21:51:50.127 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928/remote/smithi002/log 2024-03-11T21:51:50.129 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T21:51:50.267 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928/remote/smithi122/log 2024-03-11T21:51:50.268 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T21:51:50.369 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928/remote/smithi165/log 2024-03-11T21:51:50.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T21:51:50.462 INFO:tasks.cephadm:Removing cluster... 2024-03-11T21:51:50.462 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 85326acc-dff0-11ee-95c8-87774f69a715 --force 2024-03-11T21:51:50.647 INFO:teuthology.orchestra.run.smithi002.stdout:Deleting cluster with fsid: 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:51:50.762 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 85326acc-dff0-11ee-95c8-87774f69a715 --force 2024-03-11T21:51:50.941 INFO:teuthology.orchestra.run.smithi122.stdout:Deleting cluster with fsid: 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:51:51.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 85326acc-dff0-11ee-95c8-87774f69a715 --force 2024-03-11T21:51:51.233 INFO:teuthology.orchestra.run.smithi165.stdout:Deleting cluster with fsid: 85326acc-dff0-11ee-95c8-87774f69a715 2024-03-11T21:51:51.346 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T21:51:51.346 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T21:51:51.362 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T21:51:51.377 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T21:51:51.392 INFO:tasks.cephadm:Teardown complete 2024-03-11T21:51:51.393 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-11T21:51:51.403 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi002:/dev/vg_nvme/lv_1... 2024-03-11T21:51:51.404 DEBUG:teuthology.orchestra.run.smithi002:> sudo nvme disconnect -n lv_1 2024-03-11T21:51:51.554 INFO:teuthology.orchestra.run.smithi002.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T21:51:51.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:51.555 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi002:/dev/vg_nvme/lv_2... 2024-03-11T21:51:51.555 DEBUG:teuthology.orchestra.run.smithi002:> sudo nvme disconnect -n lv_2 2024-03-11T21:51:51.694 INFO:teuthology.orchestra.run.smithi002.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T21:51:51.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:51.695 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi002:/dev/vg_nvme/lv_3... 2024-03-11T21:51:51.696 DEBUG:teuthology.orchestra.run.smithi002:> sudo nvme disconnect -n lv_3 2024-03-11T21:51:51.873 INFO:teuthology.orchestra.run.smithi002.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T21:51:51.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:51.875 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi002:/dev/vg_nvme/lv_4... 2024-03-11T21:51:51.875 DEBUG:teuthology.orchestra.run.smithi002:> sudo nvme disconnect -n lv_4 2024-03-11T21:51:52.016 INFO:teuthology.orchestra.run.smithi002.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T21:51:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:52.017 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T21:51:52.017 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/scratch_devs 2024-03-11T21:51:52.044 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi122:/dev/vg_nvme/lv_1... 2024-03-11T21:51:52.045 DEBUG:teuthology.orchestra.run.smithi122:> sudo nvme disconnect -n lv_1 2024-03-11T21:51:52.201 INFO:teuthology.orchestra.run.smithi122.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T21:51:52.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:52.203 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi122:/dev/vg_nvme/lv_2... 2024-03-11T21:51:52.204 DEBUG:teuthology.orchestra.run.smithi122:> sudo nvme disconnect -n lv_2 2024-03-11T21:51:52.366 INFO:teuthology.orchestra.run.smithi122.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T21:51:52.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:52.368 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi122:/dev/vg_nvme/lv_3... 2024-03-11T21:51:52.368 DEBUG:teuthology.orchestra.run.smithi122:> sudo nvme disconnect -n lv_3 2024-03-11T21:51:52.511 INFO:teuthology.orchestra.run.smithi122.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T21:51:52.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:52.513 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi122:/dev/vg_nvme/lv_4... 2024-03-11T21:51:52.513 DEBUG:teuthology.orchestra.run.smithi122:> sudo nvme disconnect -n lv_4 2024-03-11T21:51:52.682 INFO:teuthology.orchestra.run.smithi122.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T21:51:52.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:52.684 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-11T21:51:52.684 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/scratch_devs 2024-03-11T21:51:52.711 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi165:/dev/vg_nvme/lv_1... 2024-03-11T21:51:52.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo nvme disconnect -n lv_1 2024-03-11T21:51:52.858 INFO:teuthology.orchestra.run.smithi165.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T21:51:52.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:52.860 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi165:/dev/vg_nvme/lv_2... 2024-03-11T21:51:52.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo nvme disconnect -n lv_2 2024-03-11T21:51:53.000 INFO:teuthology.orchestra.run.smithi165.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T21:51:53.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:53.002 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi165:/dev/vg_nvme/lv_3... 2024-03-11T21:51:53.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo nvme disconnect -n lv_3 2024-03-11T21:51:53.215 INFO:teuthology.orchestra.run.smithi165.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T21:51:53.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:53.217 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi165:/dev/vg_nvme/lv_4... 2024-03-11T21:51:53.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo nvme disconnect -n lv_4 2024-03-11T21:51:53.381 INFO:teuthology.orchestra.run.smithi165.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T21:51:53.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:53.384 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T21:51:53.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/scratch_devs 2024-03-11T21:51:53.421 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T21:51:53.433 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T21:51:53.433 DEBUG:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:51:53.435 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:51:53.438 DEBUG:teuthology.orchestra.run.smithi165:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:51:53.448 INFO:teuthology.orchestra.run.smithi002.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:51:53.450 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:51:53.452 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:51:53.452 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2024-03-11T21:51:53.452 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 5 +4343us[+4361us] +/- 98ms 2024-03-11T21:51:53.453 INFO:teuthology.orchestra.run.smithi002.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -2452us[-2433us] +/- 74ms 2024-03-11T21:51:53.453 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 -846us[ -827us] +/- 75ms 2024-03-11T21:51:53.453 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:51:53.454 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:51:53.454 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-03-11T21:51:53.454 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 4 +4356us[+4356us] +/- 98ms 2024-03-11T21:51:53.454 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -2431us[-2393us] +/- 74ms 2024-03-11T21:51:53.454 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 -830us[ -792us] +/- 75ms 2024-03-11T21:51:53.454 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:51:53.475 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:51:53.479 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:51:53.479 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-03-11T21:51:53.479 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +4375us[+4388us] +/- 98ms 2024-03-11T21:51:53.479 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 5 -2419us[-2419us] +/- 74ms 2024-03-11T21:51:53.479 INFO:teuthology.orchestra.run.smithi165.stdout:^* hv03.front.sepia.ceph.com 3 6 377 6 -796us[ -782us] +/- 75ms 2024-03-11T21:51:53.479 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:51:53.480 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T21:51:53.489 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T21:51:53.490 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T21:51:53.499 DEBUG:teuthology.orchestra.run.smithi002:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T21:51:53.576 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T21:51:53.644 DEBUG:teuthology.orchestra.run.smithi165:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T21:51:53.713 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T21:51:53.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:53.744 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T21:51:53.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:53.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T21:51:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:53.803 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T21:51:53.813 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T21:51:53.822 INFO:teuthology.task.internal:Duration was 721.601991 seconds 2024-03-11T21:51:53.822 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T21:51:53.831 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T21:51:53.831 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T21:51:53.834 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T21:51:53.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T21:51:53.875 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:51:53.876 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:51:53.886 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:51:54.208 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T21:51:54.209 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi002.front.sepia.ceph.com 2024-03-11T21:51:54.209 DEBUG:teuthology.orchestra.run.smithi002:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T21:51:54.229 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-03-11T21:51:54.230 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T21:51:54.276 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi165.front.sepia.ceph.com 2024-03-11T21:51:54.276 DEBUG:teuthology.orchestra.run.smithi165:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T21:51:54.317 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T21:51:54.318 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T21:51:54.320 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T21:51:54.322 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T21:51:54.386 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T21:51:54.386 DEBUG:teuthology.orchestra.run.smithi002:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T21:51:54.390 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T21:51:54.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T21:51:54.586 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T21:51:54.596 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T21:51:54.596 DEBUG:teuthology.orchestra.run.smithi002:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T21:51:54.623 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T21:51:54.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T21:51:54.678 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T21:51:54.689 DEBUG:teuthology.orchestra.run.smithi002:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:51:54.691 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:51:54.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:51:54.714 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = core 2024-03-11T21:51:54.718 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-03-11T21:51:54.744 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = core 2024-03-11T21:51:54.764 DEBUG:teuthology.orchestra.run.smithi002:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:51:54.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:54.792 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:51:54.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:54.806 DEBUG:teuthology.orchestra.run.smithi165:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:51:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:54.820 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T21:51:54.831 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T21:51:54.831 DEBUG:teuthology.misc:Transferring archived files from smithi002:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928/remote/smithi002 2024-03-11T21:51:54.832 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T21:51:54.870 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928/remote/smithi122 2024-03-11T21:51:54.870 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T21:51:54.907 DEBUG:teuthology.misc:Transferring archived files from smithi165:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928/remote/smithi165 2024-03-11T21:51:54.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T21:51:54.945 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T21:51:54.946 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T21:51:54.948 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T21:51:54.950 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T21:51:54.995 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T21:51:55.005 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T21:51:55.005 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T21:51:55.014 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T21:51:55.015 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T21:51:55.017 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T21:51:55.019 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T21:51:55.028 INFO:teuthology.orchestra.run.smithi002.stdout: 265841 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 21:51 /home/ubuntu/cephtest 2024-03-11T21:51:55.031 INFO:teuthology.orchestra.run.smithi122.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 21:51 /home/ubuntu/cephtest 2024-03-11T21:51:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 21:51 /home/ubuntu/cephtest 2024-03-11T21:51:55.051 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T21:51:55.067 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T21:51:55.093 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T21:51:55.122 DEBUG:teuthology.nuke:shortname: smithi002 2024-03-11T21:51:55.122 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:51:55.149 DEBUG:teuthology.nuke:shortname: smithi122 2024-03-11T21:51:55.149 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:51:55.154 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:30:48.054061', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMyy1VnFJ1yzz4Hmpu/kPgfDydhNoluk0E+8C0lV7fgx1Xb657od08p7Wg9G6KJ0QFtpqyx7HLiwEQstPF/jCm4='} 2024-03-11T21:51:55.173 DEBUG:teuthology.nuke:shortname: smithi165 2024-03-11T21:51:55.174 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:51:55.180 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:30:48.056839', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOrxgDQKCS1btsbAGFldJn7lmKH+Ph15uGCOQQQh+JkJNirOwLCtFIgLf0iymgkt6lig90ScYBRO3jqYH7njFW4='} 2024-03-11T21:51:55.198 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591928', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:30:48.058597', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGPIaeiX8fSXrD2FB9fHS58n7cqRRjH7y6geYjdDoU/1nykpxDVYi6daHwhsPzUUtBxCGd1yWQbz2Ym3OXhoBfI='} 2024-03-11T21:51:55.209 INFO:teuthology.orchestra.console.smithi002:Power off 2024-03-11T21:51:55.210 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T21:51:55.241 DEBUG:teuthology.orchestra.console.smithi002:power off output: Chassis Power Control: Down/Off 2024-03-11T21:51:55.241 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:51:55.258 INFO:teuthology.orchestra.console.smithi165:Power off 2024-03-11T21:51:55.258 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T21:51:55.271 INFO:teuthology.orchestra.console.smithi122:Power off 2024-03-11T21:51:55.271 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T21:51:55.281 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2024-03-11T21:51:55.284 DEBUG:teuthology.orchestra.console.smithi165:power off output: Chassis Power Control: Down/Off 2024-03-11T21:51:55.284 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:51:55.297 DEBUG:teuthology.orchestra.console.smithi122:power off output: Chassis Power Control: Down/Off 2024-03-11T21:51:55.297 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:51:55.308 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-03-11T21:51:55.321 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-03-11T21:51:59.282 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:51:59.309 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:51:59.322 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:51:59.393 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2024-03-11T21:51:59.419 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-03-11T21:51:59.433 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-03-11T21:52:03.394 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:52:03.420 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:52:03.434 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:52:03.506 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is off 2024-03-11T21:52:03.533 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-03-11T21:52:03.544 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is off 2024-03-11T21:52:03.608 INFO:teuthology.orchestra.console.smithi002:Power off completed 2024-03-11T21:52:03.645 INFO:teuthology.orchestra.console.smithi122:Power off completed 2024-03-11T21:52:07.536 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:52:07.645 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is off 2024-03-11T21:52:07.746 INFO:teuthology.orchestra.console.smithi165:Power off completed 2024-03-11T21:52:07.850 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} duration: 721.6019909381866 failure_reason: '"2024-03-11T21:47:43.460833+0000 mon.a (mon.0) 248 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-11T21:52:07.850 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T21:52:07.935 INFO:teuthology.run:FAIL