2024-03-04T15:18:32.823 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-03-04T15:18:32.823 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216 --description orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} --name tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi --verbose --owner scheduled_tonay@teuthology -- /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216/orig.config.yaml 2024-03-04T15:18:32.851 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T15:18:32.937 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216 branch: wip-asm0deuz-testing-2024-03-04-1211 description: orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} email: tonay@ibm.com first_in_suite: false job_id: '7581216' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-asm0deuz-testing-2024-03-04-1211 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-asm0deuz-testing-2024-03-04-1211 sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 owner: scheduled_tonay@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 3905 sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 sleep_before_teardown: 0 subset: 1/14 suite: orch:cephadm suite_branch: wip-asm0deuz-testing-2024-03-04-1211 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 targets: smithi005.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAEwl8LNvr8Q+ASGvGG4IArMD4y7zEJYEF3RiJCGvEEMSIZC4whN+b1tyKctc8owViKgIZEVu21QpHX7U07jn7Q= smithi080.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGKJoDJ0ZThG8/SoNJ8S1EiuczPEDkJyIH/ICdHs9wI/Jkz5UvvKzjew480GSGpdPzp1tpLKdk/qLfYuo/6yDuc= smithi087.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHg2B34znpP0NOSpnSvpFgaQ5yUZInWSIRBio/OYMkB7Qb+6VQzc37lG9FHlYLhXGyfULbVP9HeVuss9lnsj2S0= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-03-04_15:07:38 tube: smithi user: tonay verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-04T15:18:32.937 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa; will attempt to use it 2024-03-04T15:18:32.937 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa/tasks 2024-03-04T15:18:32.938 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-04T15:18:32.939 INFO:teuthology.task.internal:Checking packages... 2024-03-04T15:18:32.963 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9ca0b5de35ed230e84387abdd8e619ed8e4f3282' 2024-03-04T15:18:32.963 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-04T15:18:32.963 INFO:teuthology.packaging:ref: None 2024-03-04T15:18:32.963 INFO:teuthology.packaging:tag: None 2024-03-04T15:18:32.963 INFO:teuthology.packaging:branch: wip-asm0deuz-testing-2024-03-04-1211 2024-03-04T15:18:32.963 INFO:teuthology.packaging:sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:18:32.963 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-asm0deuz-testing-2024-03-04-1211 2024-03-04T15:18:33.111 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1826.g9ca0b5de 2024-03-04T15:18:33.112 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-04T15:18:33.119 INFO:teuthology.task.internal:no buildpackages task found 2024-03-04T15:18:33.119 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-04T15:18:33.126 INFO:teuthology.task.internal:Saving configuration 2024-03-04T15:18:33.137 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-04T15:18:33.142 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T15:18:33.177 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:09:16.519420', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAEwl8LNvr8Q+ASGvGG4IArMD4y7zEJYEF3RiJCGvEEMSIZC4whN+b1tyKctc8owViKgIZEVu21QpHX7U07jn7Q='} 2024-03-04T15:18:33.208 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi080.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:09:16.517521', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGKJoDJ0ZThG8/SoNJ8S1EiuczPEDkJyIH/ICdHs9wI/Jkz5UvvKzjew480GSGpdPzp1tpLKdk/qLfYuo/6yDuc='} 2024-03-04T15:18:33.241 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:09:16.521161', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHg2B34znpP0NOSpnSvpFgaQ5yUZInWSIRBio/OYMkB7Qb+6VQzc37lG9FHlYLhXGyfULbVP9HeVuss9lnsj2S0='} 2024-03-04T15:18:33.241 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-04T15:18:33.247 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-04T15:18:33.247 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-04T15:18:33.247 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-04T15:18:33.248 INFO:teuthology.run_tasks:Running task console_log... 2024-03-04T15:18:33.373 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3cdf61df70>, signals=[15]) 2024-03-04T15:18:33.373 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-04T15:18:33.381 INFO:teuthology.task.internal:Opening connections... 2024-03-04T15:18:33.381 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2024-03-04T15:18:33.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:18:33.497 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi080.front.sepia.ceph.com 2024-03-04T15:18:33.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:18:33.594 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2024-03-04T15:18:33.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:18:33.688 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-04T15:18:33.695 DEBUG:teuthology.orchestra.run.smithi005:> uname -m 2024-03-04T15:18:33.712 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2024-03-04T15:18:33.713 DEBUG:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2024-03-04T15:18:33.769 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="CentOS Stream" 2024-03-04T15:18:33.769 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="9" 2024-03-04T15:18:33.769 INFO:teuthology.orchestra.run.smithi005.stdout:ID="centos" 2024-03-04T15:18:33.769 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE="rhel fedora" 2024-03-04T15:18:33.769 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="9" 2024-03-04T15:18:33.769 INFO:teuthology.orchestra.run.smithi005.stdout:PLATFORM_ID="platform:el9" 2024-03-04T15:18:33.769 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T15:18:33.770 INFO:teuthology.orchestra.run.smithi005.stdout:ANSI_COLOR="0;31" 2024-03-04T15:18:33.770 INFO:teuthology.orchestra.run.smithi005.stdout:LOGO="fedora-logo-icon" 2024-03-04T15:18:33.770 INFO:teuthology.orchestra.run.smithi005.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T15:18:33.770 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://centos.org/" 2024-03-04T15:18:33.770 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T15:18:33.770 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T15:18:33.770 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T15:18:33.771 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2024-03-04T15:18:33.799 DEBUG:teuthology.orchestra.run.smithi080:> uname -m 2024-03-04T15:18:33.816 INFO:teuthology.orchestra.run.smithi080.stdout:x86_64 2024-03-04T15:18:33.817 DEBUG:teuthology.orchestra.run.smithi080:> cat /etc/os-release 2024-03-04T15:18:33.873 INFO:teuthology.orchestra.run.smithi080.stdout:NAME="CentOS Stream" 2024-03-04T15:18:33.873 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION="9" 2024-03-04T15:18:33.873 INFO:teuthology.orchestra.run.smithi080.stdout:ID="centos" 2024-03-04T15:18:33.873 INFO:teuthology.orchestra.run.smithi080.stdout:ID_LIKE="rhel fedora" 2024-03-04T15:18:33.873 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_ID="9" 2024-03-04T15:18:33.874 INFO:teuthology.orchestra.run.smithi080.stdout:PLATFORM_ID="platform:el9" 2024-03-04T15:18:33.874 INFO:teuthology.orchestra.run.smithi080.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T15:18:33.874 INFO:teuthology.orchestra.run.smithi080.stdout:ANSI_COLOR="0;31" 2024-03-04T15:18:33.874 INFO:teuthology.orchestra.run.smithi080.stdout:LOGO="fedora-logo-icon" 2024-03-04T15:18:33.874 INFO:teuthology.orchestra.run.smithi080.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T15:18:33.874 INFO:teuthology.orchestra.run.smithi080.stdout:HOME_URL="https://centos.org/" 2024-03-04T15:18:33.874 INFO:teuthology.orchestra.run.smithi080.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T15:18:33.874 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T15:18:33.874 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T15:18:33.874 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server 2024-03-04T15:18:33.899 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2024-03-04T15:18:33.916 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2024-03-04T15:18:33.917 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2024-03-04T15:18:33.973 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2024-03-04T15:18:33.973 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="9" 2024-03-04T15:18:33.974 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2024-03-04T15:18:33.974 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2024-03-04T15:18:33.974 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="9" 2024-03-04T15:18:33.974 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el9" 2024-03-04T15:18:33.974 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T15:18:33.974 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2024-03-04T15:18:33.974 INFO:teuthology.orchestra.run.smithi087.stdout:LOGO="fedora-logo-icon" 2024-03-04T15:18:33.974 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T15:18:33.974 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2024-03-04T15:18:33.974 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T15:18:33.974 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T15:18:33.974 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T15:18:33.974 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2024-03-04T15:18:34.007 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-04T15:18:34.016 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-04T15:18:34.021 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-04T15:18:34.021 DEBUG:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2024-03-04T15:18:34.024 DEBUG:teuthology.orchestra.run.smithi080:> test '!' -e /home/ubuntu/cephtest 2024-03-04T15:18:34.026 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2024-03-04T15:18:34.042 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-04T15:18:34.049 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-04T15:18:34.049 DEBUG:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2024-03-04T15:18:34.082 DEBUG:teuthology.orchestra.run.smithi080:> test -z $(ls -A /var/lib/ceph) 2024-03-04T15:18:34.084 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2024-03-04T15:18:34.178 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-04T15:18:34.318 INFO:teuthology.run_tasks:Running task kernel... 2024-03-04T15:18:34.332 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-04T15:18:34.332 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-04T15:18:34.332 DEBUG:teuthology.orchestra.run.smithi005:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T15:18:34.333 DEBUG:teuthology.orchestra.run.smithi080:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T15:18:34.333 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T15:18:34.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:18:34.349 DEBUG:teuthology.orchestra.run.smithi080:> uname -r 2024-03-04T15:18:34.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:18:34.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:18:34.351 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-03-04T15:18:34.352 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-03-04T15:18:34.407 INFO:teuthology.orchestra.run.smithi080.stdout:5.14.0-361.el9.x86_64 2024-03-04T15:18:34.407 INFO:teuthology.task.kernel:Running kernel on smithi080: 5.14.0-361.el9.x86_64 2024-03-04T15:18:34.407 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum install -y kernel 2024-03-04T15:18:34.409 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-361.el9.x86_64 2024-03-04T15:18:34.409 INFO:teuthology.task.kernel:Running kernel on smithi087: 5.14.0-361.el9.x86_64 2024-03-04T15:18:34.409 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum install -y kernel 2024-03-04T15:18:34.410 INFO:teuthology.orchestra.run.smithi005.stdout:5.14.0-361.el9.x86_64 2024-03-04T15:18:34.410 INFO:teuthology.task.kernel:Running kernel on smithi005: 5.14.0-361.el9.x86_64 2024-03-04T15:18:34.410 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum install -y kernel 2024-03-04T15:18:36.852 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-9 - AppStream 8.6 MB/s | 6.7 MB 00:00 2024-03-04T15:18:36.885 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-9 - AppStream 8.4 MB/s | 6.7 MB 00:00 2024-03-04T15:18:36.961 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-9 - AppStream 8.3 MB/s | 6.7 MB 00:00 2024-03-04T15:18:38.452 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-04T15:18:38.477 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-04T15:18:38.552 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-04T15:18:39.237 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-04T15:18:39.371 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-04T15:18:39.520 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-04T15:18:41.344 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-04T15:18:41.736 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-04T15:18:41.744 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-04T15:18:45.405 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-04T15:18:45.755 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-04T15:18:46.106 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-04T15:18:46.853 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 15 kB 00:00 2024-03-04T15:18:47.137 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 15 kB 00:00 2024-03-04T15:18:47.546 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - Extras packages 87 kB/s | 15 kB 00:00 2024-03-04T15:18:48.449 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2024-03-04T15:18:48.563 INFO:teuthology.orchestra.run.smithi080.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2024-03-04T15:18:49.422 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2024-03-04T15:18:53.363 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-04T15:18:53.421 INFO:teuthology.orchestra.run.smithi080.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-03-04T15:18:54.539 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-03-04T15:18:55.570 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T15:18:55.570 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T15:18:55.661 INFO:teuthology.orchestra.run.smithi080.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T15:18:55.661 INFO:teuthology.orchestra.run.smithi080.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T15:18:55.669 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-04T15:18:55.681 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-04T15:18:55.681 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-03-04T15:18:55.681 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-04T15:18:55.681 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-04T15:18:55.681 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T15:18:55.681 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-03-04T15:18:55.681 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T15:18:55.682 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T15:18:55.682 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T15:18:55.682 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-04T15:18:55.682 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-04T15:18:55.682 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-04T15:18:55.682 INFO:teuthology.orchestra.run.smithi087.stdout:Install 4 Packages 2024-03-04T15:18:55.682 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-04T15:18:55.682 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 100 M 2024-03-04T15:18:55.683 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 123 M 2024-03-04T15:18:55.683 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-04T15:18:55.740 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-04T15:18:55.749 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-04T15:18:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-03-04T15:18:55.749 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-04T15:18:55.749 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-04T15:18:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T15:18:55.749 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-03-04T15:18:55.749 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T15:18:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T15:18:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T15:18:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-04T15:18:55.750 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-04T15:18:55.750 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-04T15:18:55.750 INFO:teuthology.orchestra.run.smithi080.stdout:Install 4 Packages 2024-03-04T15:18:55.750 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-04T15:18:55.751 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 100 M 2024-03-04T15:18:55.751 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 123 M 2024-03-04T15:18:55.751 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-04T15:18:56.106 INFO:teuthology.orchestra.run.smithi087.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 24 MB/s | 6.0 MB 00:00 2024-03-04T15:18:56.181 INFO:teuthology.orchestra.run.smithi080.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 21 MB/s | 6.0 MB 00:00 2024-03-04T15:18:56.415 INFO:teuthology.orchestra.run.smithi087.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 37 MB/s | 21 MB 00:00 2024-03-04T15:18:56.473 INFO:teuthology.orchestra.run.smithi080.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 36 MB/s | 21 MB 00:00 2024-03-04T15:18:56.840 INFO:teuthology.orchestra.run.smithi087.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 40 MB/s | 39 MB 00:00 2024-03-04T15:18:56.862 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T15:18:56.862 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T15:18:56.881 INFO:teuthology.orchestra.run.smithi080.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 48 MB/s | 33 MB 00:00 2024-03-04T15:18:56.952 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-04T15:18:56.960 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-04T15:18:56.960 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-03-04T15:18:56.960 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-04T15:18:56.960 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-04T15:18:56.960 INFO:teuthology.orchestra.run.smithi005.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T15:18:56.960 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-03-04T15:18:56.960 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T15:18:56.960 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T15:18:56.960 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T15:18:56.960 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:18:56.961 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-04T15:18:56.961 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-04T15:18:56.961 INFO:teuthology.orchestra.run.smithi005.stdout:Install 4 Packages 2024-03-04T15:18:56.961 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:18:56.961 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 100 M 2024-03-04T15:18:56.961 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 123 M 2024-03-04T15:18:56.961 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-04T15:18:57.149 INFO:teuthology.orchestra.run.smithi087.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 32 MB/s | 33 MB 00:01 2024-03-04T15:18:57.152 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:18:57.152 INFO:teuthology.orchestra.run.smithi087.stdout:Total 68 MB/s | 100 MB 00:01 2024-03-04T15:18:57.307 INFO:teuthology.orchestra.run.smithi005.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 21 MB/s | 6.0 MB 00:00 2024-03-04T15:18:57.314 INFO:teuthology.orchestra.run.smithi080.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 28 MB/s | 39 MB 00:01 2024-03-04T15:18:57.318 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:18:57.318 INFO:teuthology.orchestra.run.smithi080.stdout:Total 63 MB/s | 100 MB 00:01 2024-03-04T15:18:57.582 INFO:teuthology.orchestra.run.smithi005.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 37 MB/s | 21 MB 00:00 2024-03-04T15:18:57.957 INFO:teuthology.orchestra.run.smithi005.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 52 MB/s | 33 MB 00:00 2024-03-04T15:18:58.040 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-04T15:18:58.205 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-04T15:18:58.357 INFO:teuthology.orchestra.run.smithi005.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 29 MB/s | 39 MB 00:01 2024-03-04T15:18:58.360 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:18:58.361 INFO:teuthology.orchestra.run.smithi005.stdout:Total 71 MB/s | 100 MB 00:01 2024-03-04T15:18:58.408 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-04T15:18:58.409 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-04T15:18:58.554 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-04T15:18:58.554 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-04T15:18:59.285 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-04T15:18:59.703 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-04T15:18:59.703 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-04T15:19:00.031 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-04T15:19:00.031 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-04T15:19:00.232 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-04T15:19:00.233 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-04T15:19:01.143 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-04T15:19:01.144 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-04T15:19:02.751 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-04T15:19:02.943 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-04T15:19:03.421 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:19:03.558 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:19:03.598 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:19:03.718 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:19:04.010 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-04T15:19:04.697 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:19:04.832 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:19:06.329 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:19:06.495 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:19:06.503 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:19:06.696 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:19:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:19:07.917 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:19:11.241 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:19:11.371 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:11.416 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:19:11.839 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:13.044 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:19:13.156 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:16.132 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:16.589 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:18.315 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:45.476 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:45.477 INFO:teuthology.orchestra.run.smithi087.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T15:19:45.477 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-04T15:19:45.561 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:45.785 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:45.786 INFO:teuthology.orchestra.run.smithi080.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T15:19:45.786 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-04T15:19:45.871 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:46.024 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:46.024 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:19:46.024 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:19:46.024 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:19:46.282 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:46.283 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:19:46.283 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:19:46.283 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:19:46.445 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:46.445 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-04T15:19:46.445 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-04T15:19:46.445 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T15:19:46.446 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T15:19:46.446 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T15:19:46.446 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T15:19:46.446 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-04T15:19:46.446 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-04T15:19:46.704 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:46.704 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-04T15:19:46.704 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-04T15:19:46.704 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T15:19:46.704 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T15:19:46.705 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T15:19:46.705 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T15:19:46.705 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-04T15:19:46.705 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-04T15:19:46.978 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:19:47.116 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:19:47.117 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:19:47.117 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T15:19:47.117 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T15:19:47.117 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T15:19:47.117 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-04T15:19:47.117 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi087.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T15:19:47.117 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:19:47.237 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:19:47.238 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:19:47.238 DEBUG:teuthology.orchestra.run.smithi087:> sudo rpm -qi grub2-tools 2024-03-04T15:19:47.239 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:19:47.271 INFO:teuthology.orchestra.run.smithi087.stdout:Name : grub2-tools 2024-03-04T15:19:47.271 INFO:teuthology.orchestra.run.smithi087.stdout:Epoch : 1 2024-03-04T15:19:47.271 INFO:teuthology.orchestra.run.smithi087.stdout:Version : 2.06 2024-03-04T15:19:47.271 INFO:teuthology.orchestra.run.smithi087.stdout:Release : 68.el9 2024-03-04T15:19:47.272 INFO:teuthology.orchestra.run.smithi087.stdout:Architecture: x86_64 2024-03-04T15:19:47.272 INFO:teuthology.orchestra.run.smithi087.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T15:19:47.272 INFO:teuthology.orchestra.run.smithi087.stdout:Group : Unspecified 2024-03-04T15:19:47.272 INFO:teuthology.orchestra.run.smithi087.stdout:Size : 8264254 2024-03-04T15:19:47.272 INFO:teuthology.orchestra.run.smithi087.stdout:License : GPLv3+ 2024-03-04T15:19:47.272 INFO:teuthology.orchestra.run.smithi087.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T15:19:47.272 INFO:teuthology.orchestra.run.smithi087.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T15:19:47.272 INFO:teuthology.orchestra.run.smithi087.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T15:19:47.272 INFO:teuthology.orchestra.run.smithi087.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T15:19:47.272 INFO:teuthology.orchestra.run.smithi087.stdout:Packager : builder@centos.org 2024-03-04T15:19:47.272 INFO:teuthology.orchestra.run.smithi087.stdout:Vendor : CentOS 2024-03-04T15:19:47.272 INFO:teuthology.orchestra.run.smithi087.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T15:19:47.272 INFO:teuthology.orchestra.run.smithi087.stdout:Summary : Support tools for GRUB. 2024-03-04T15:19:47.272 INFO:teuthology.orchestra.run.smithi087.stdout:Description : 2024-03-04T15:19:47.272 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-04T15:19:47.273 INFO:teuthology.orchestra.run.smithi087.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T15:19:47.273 INFO:teuthology.orchestra.run.smithi087.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T15:19:47.273 INFO:teuthology.orchestra.run.smithi087.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T15:19:47.273 INFO:teuthology.orchestra.run.smithi087.stdout:hardware devices. 2024-03-04T15:19:47.273 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-04T15:19:47.273 INFO:teuthology.orchestra.run.smithi087.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T15:19:47.274 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T15:19:47.274 INFO:teuthology.task.kernel:Updating grub on smithi087 to boot 5.14.0-427.el9.x86_64 2024-03-04T15:19:47.274 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T15:19:47.404 INFO:teuthology.orchestra.run.smithi080.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:19:47.404 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:19:47.404 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T15:19:47.404 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T15:19:47.404 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T15:19:47.404 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-04T15:19:47.404 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi080.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T15:19:47.405 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:19:47.524 INFO:teuthology.orchestra.run.smithi080.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:19:47.525 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:19:47.525 DEBUG:teuthology.orchestra.run.smithi080:> sudo rpm -qi grub2-tools 2024-03-04T15:19:47.559 INFO:teuthology.orchestra.run.smithi080.stdout:Name : grub2-tools 2024-03-04T15:19:47.560 INFO:teuthology.orchestra.run.smithi080.stdout:Epoch : 1 2024-03-04T15:19:47.560 INFO:teuthology.orchestra.run.smithi080.stdout:Version : 2.06 2024-03-04T15:19:47.560 INFO:teuthology.orchestra.run.smithi080.stdout:Release : 68.el9 2024-03-04T15:19:47.560 INFO:teuthology.orchestra.run.smithi080.stdout:Architecture: x86_64 2024-03-04T15:19:47.560 INFO:teuthology.orchestra.run.smithi080.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T15:19:47.560 INFO:teuthology.orchestra.run.smithi080.stdout:Group : Unspecified 2024-03-04T15:19:47.560 INFO:teuthology.orchestra.run.smithi080.stdout:Size : 8264254 2024-03-04T15:19:47.561 INFO:teuthology.orchestra.run.smithi080.stdout:License : GPLv3+ 2024-03-04T15:19:47.561 INFO:teuthology.orchestra.run.smithi080.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T15:19:47.561 INFO:teuthology.orchestra.run.smithi080.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T15:19:47.561 INFO:teuthology.orchestra.run.smithi080.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T15:19:47.561 INFO:teuthology.orchestra.run.smithi080.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T15:19:47.561 INFO:teuthology.orchestra.run.smithi080.stdout:Packager : builder@centos.org 2024-03-04T15:19:47.561 INFO:teuthology.orchestra.run.smithi080.stdout:Vendor : CentOS 2024-03-04T15:19:47.561 INFO:teuthology.orchestra.run.smithi080.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T15:19:47.561 INFO:teuthology.orchestra.run.smithi080.stdout:Summary : Support tools for GRUB. 2024-03-04T15:19:47.561 INFO:teuthology.orchestra.run.smithi080.stdout:Description : 2024-03-04T15:19:47.562 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-04T15:19:47.562 INFO:teuthology.orchestra.run.smithi080.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T15:19:47.562 INFO:teuthology.orchestra.run.smithi080.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T15:19:47.562 INFO:teuthology.orchestra.run.smithi080.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T15:19:47.562 INFO:teuthology.orchestra.run.smithi080.stdout:hardware devices. 2024-03-04T15:19:47.562 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-04T15:19:47.562 INFO:teuthology.orchestra.run.smithi080.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T15:19:47.563 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T15:19:47.564 INFO:teuthology.task.kernel:Updating grub on smithi080 to boot 5.14.0-427.el9.x86_64 2024-03-04T15:19:47.564 DEBUG:teuthology.orchestra.run.smithi080:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T15:19:47.891 INFO:teuthology.orchestra.run.smithi087.stderr:Generating grub configuration file ... 2024-03-04T15:19:48.192 INFO:teuthology.orchestra.run.smithi080.stderr:Generating grub configuration file ... 2024-03-04T15:19:49.058 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:49.058 INFO:teuthology.orchestra.run.smithi005.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T15:19:49.058 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:19:49.145 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:49.163 INFO:teuthology.orchestra.run.smithi087.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T15:19:49.197 INFO:teuthology.orchestra.run.smithi087.stderr:done 2024-03-04T15:19:49.199 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-03-04T15:19:49.214 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.wZUoaFRK81 2024-03-04T15:19:49.215 DEBUG:teuthology.orchestra.run.smithi087:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wZUoaFRK81 2024-03-04T15:19:49.286 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.wZUoaFRK81 2024-03-04T15:19:49.436 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.wZUoaFRK81 is 6KB 2024-03-04T15:19:49.485 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.wZUoaFRK81 2024-03-04T15:19:49.500 DEBUG:teuthology.orchestra.run.smithi087:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T15:19:49.565 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T15:19:49.565 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T15:19:49.565 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T15:19:49.565 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T15:19:49.566 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T15:19:49.591 INFO:teuthology.orchestra.run.smithi080.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T15:19:49.625 INFO:teuthology.orchestra.run.smithi080.stderr:done 2024-03-04T15:19:49.628 DEBUG:teuthology.orchestra.run.smithi080:> mktemp 2024-03-04T15:19:49.645 INFO:teuthology.orchestra.run.smithi080.stdout:/tmp/tmp.wR1xMvItpm 2024-03-04T15:19:49.645 DEBUG:teuthology.orchestra.run.smithi080:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wR1xMvItpm 2024-03-04T15:19:49.650 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:49.650 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:19:49.650 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:19:49.651 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:19:49.701 DEBUG:teuthology.orchestra.run.smithi087:> sudo shutdown -r now 2024-03-04T15:19:49.713 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0666 /tmp/tmp.wR1xMvItpm 2024-03-04T15:19:49.844 DEBUG:teuthology.orchestra.remote:smithi080:/tmp/tmp.wR1xMvItpm is 6KB 2024-03-04T15:19:49.896 DEBUG:teuthology.orchestra.run.smithi080:> rm -fr /tmp/tmp.wR1xMvItpm 2024-03-04T15:19:49.910 DEBUG:teuthology.orchestra.run.smithi080:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T15:19:49.984 INFO:teuthology.orchestra.run.smithi080.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T15:19:49.984 INFO:teuthology.orchestra.run.smithi080.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T15:19:49.984 INFO:teuthology.orchestra.run.smithi080.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T15:19:49.984 INFO:teuthology.orchestra.run.smithi080.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T15:19:49.985 DEBUG:teuthology.orchestra.run.smithi080:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T15:19:50.071 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:19:50.072 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:19:50.072 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-04T15:19:50.072 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T15:19:50.072 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T15:19:50.072 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T15:19:50.072 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T15:19:50.072 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:19:50.072 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-04T15:19:50.142 DEBUG:teuthology.orchestra.run.smithi080:> sudo shutdown -r now 2024-03-04T15:19:50.605 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:19:50.762 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:19:50.763 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:19:50.763 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T15:19:50.763 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T15:19:50.763 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T15:19:50.763 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-04T15:19:50.763 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi005.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T15:19:50.763 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:19:50.890 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:19:50.890 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:19:50.890 DEBUG:teuthology.orchestra.run.smithi005:> sudo rpm -qi grub2-tools 2024-03-04T15:19:50.928 INFO:teuthology.orchestra.run.smithi005.stdout:Name : grub2-tools 2024-03-04T15:19:50.928 INFO:teuthology.orchestra.run.smithi005.stdout:Epoch : 1 2024-03-04T15:19:50.928 INFO:teuthology.orchestra.run.smithi005.stdout:Version : 2.06 2024-03-04T15:19:50.928 INFO:teuthology.orchestra.run.smithi005.stdout:Release : 68.el9 2024-03-04T15:19:50.928 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: x86_64 2024-03-04T15:19:50.928 INFO:teuthology.orchestra.run.smithi005.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T15:19:50.928 INFO:teuthology.orchestra.run.smithi005.stdout:Group : Unspecified 2024-03-04T15:19:50.928 INFO:teuthology.orchestra.run.smithi005.stdout:Size : 8264254 2024-03-04T15:19:50.929 INFO:teuthology.orchestra.run.smithi005.stdout:License : GPLv3+ 2024-03-04T15:19:50.929 INFO:teuthology.orchestra.run.smithi005.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T15:19:50.929 INFO:teuthology.orchestra.run.smithi005.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T15:19:50.929 INFO:teuthology.orchestra.run.smithi005.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T15:19:50.929 INFO:teuthology.orchestra.run.smithi005.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T15:19:50.929 INFO:teuthology.orchestra.run.smithi005.stdout:Packager : builder@centos.org 2024-03-04T15:19:50.929 INFO:teuthology.orchestra.run.smithi005.stdout:Vendor : CentOS 2024-03-04T15:19:50.929 INFO:teuthology.orchestra.run.smithi005.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T15:19:50.929 INFO:teuthology.orchestra.run.smithi005.stdout:Summary : Support tools for GRUB. 2024-03-04T15:19:50.929 INFO:teuthology.orchestra.run.smithi005.stdout:Description : 2024-03-04T15:19:50.929 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:19:50.929 INFO:teuthology.orchestra.run.smithi005.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T15:19:50.929 INFO:teuthology.orchestra.run.smithi005.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T15:19:50.930 INFO:teuthology.orchestra.run.smithi005.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T15:19:50.930 INFO:teuthology.orchestra.run.smithi005.stdout:hardware devices. 2024-03-04T15:19:50.930 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:19:50.930 INFO:teuthology.orchestra.run.smithi005.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T15:19:50.931 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T15:19:50.931 INFO:teuthology.task.kernel:Updating grub on smithi005 to boot 5.14.0-427.el9.x86_64 2024-03-04T15:19:50.931 DEBUG:teuthology.orchestra.run.smithi005:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T15:19:51.538 INFO:teuthology.orchestra.run.smithi005.stderr:Generating grub configuration file ... 2024-03-04T15:19:52.832 INFO:teuthology.orchestra.run.smithi005.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T15:19:52.864 INFO:teuthology.orchestra.run.smithi005.stderr:done 2024-03-04T15:19:52.867 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-03-04T15:19:52.882 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.J3U3W9qetW 2024-03-04T15:19:52.882 DEBUG:teuthology.orchestra.run.smithi005:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.J3U3W9qetW 2024-03-04T15:19:52.954 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.J3U3W9qetW 2024-03-04T15:19:53.101 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.J3U3W9qetW is 6KB 2024-03-04T15:19:53.150 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.J3U3W9qetW 2024-03-04T15:19:53.165 DEBUG:teuthology.orchestra.run.smithi005:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T15:19:53.235 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T15:19:53.235 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T15:19:53.235 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T15:19:53.235 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T15:19:53.237 DEBUG:teuthology.orchestra.run.smithi005:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T15:19:53.399 DEBUG:teuthology.orchestra.run.smithi005:> sudo shutdown -r now 2024-03-04T15:20:19.728 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-04T15:20:19.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-03-04T15:20:19.729 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:20:20.145 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-04T15:20:20.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-04T15:20:20.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:20:23.404 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-04T15:20:23.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-04T15:20:23.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:21:19.796 DEBUG:teuthology.orchestra.remote:timed out 2024-03-04T15:21:20.186 DEBUG:teuthology.orchestra.remote:timed out 2024-03-04T15:21:23.416 DEBUG:teuthology.orchestra.remote:timed out 2024-03-04T15:21:28.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-03-04T15:21:28.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:21:29.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-04T15:21:29.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:21:32.242 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-03-04T15:21:32.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-04T15:21:32.417 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:21:32.639 DEBUG:teuthology.orchestra.run.smithi080:> true 2024-03-04T15:21:32.679 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-03-04T15:21:32.679 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-427.el9.x86_64"... 2024-03-04T15:21:32.679 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-03-04T15:21:32.694 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-427.el9.x86_64 2024-03-04T15:21:32.694 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T15:21:32.694 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T15:21:32.694 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T15:21:32.800 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-03-04T15:21:33.172 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-04T15:21:33.172 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-04T15:21:33.172 DEBUG:teuthology.orchestra.run.smithi080:> uname -r 2024-03-04T15:21:33.186 INFO:teuthology.orchestra.run.smithi080.stdout:5.14.0-427.el9.x86_64 2024-03-04T15:21:33.186 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T15:21:33.186 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T15:21:33.186 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T15:21:33.223 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-04T15:21:33.223 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-04T15:21:33.223 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-03-04T15:21:33.237 INFO:teuthology.orchestra.run.smithi005.stdout:5.14.0-427.el9.x86_64 2024-03-04T15:21:33.237 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T15:21:33.237 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T15:21:33.237 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T15:21:33.695 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-04T15:21:33.695 DEBUG:teuthology.orchestra.run.smithi087:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T15:21:33.748 INFO:teuthology.orchestra.run.smithi087.stdout:ttyS1 2024-03-04T15:21:33.783 DEBUG:teuthology.parallel:result is None 2024-03-04T15:21:34.188 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-04T15:21:34.188 DEBUG:teuthology.orchestra.run.smithi080:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T15:21:34.238 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-04T15:21:34.239 DEBUG:teuthology.orchestra.run.smithi005:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T15:21:34.288 INFO:teuthology.orchestra.run.smithi080.stdout:ttyS1 2024-03-04T15:21:34.307 INFO:teuthology.orchestra.run.smithi005.stdout:ttyS1 2024-03-04T15:21:34.330 DEBUG:teuthology.parallel:result is None 2024-03-04T15:21:34.351 DEBUG:teuthology.parallel:result is None 2024-03-04T15:21:34.351 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-04T15:21:34.358 INFO:teuthology.task.internal:Creating test directory... 2024-03-04T15:21:34.358 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T15:21:34.361 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T15:21:34.363 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T15:21:34.378 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-04T15:21:34.385 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-04T15:21:34.431 INFO:teuthology.task.internal:Creating archive directory... 2024-03-04T15:21:34.431 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T15:21:34.434 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T15:21:34.437 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T15:21:34.482 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-04T15:21:34.489 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-04T15:21:34.489 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T15:21:34.523 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T15:21:34.526 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T15:21:34.547 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:21:34.555 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:21:34.560 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:21:34.567 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:21:34.569 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:21:34.580 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:21:34.582 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-04T15:21:34.589 INFO:teuthology.task.internal:Configuring sudo... 2024-03-04T15:21:34.590 DEBUG:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T15:21:34.611 DEBUG:teuthology.orchestra.run.smithi080:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T15:21:34.624 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T15:21:34.653 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-04T15:21:34.663 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-04T15:21:34.663 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T15:21:34.682 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T15:21:34.696 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T15:21:34.711 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:21:34.791 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:21:34.833 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:21:34.929 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:21:34.964 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-04T15:21:34.964 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T15:21:35.033 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:21:35.089 DEBUG:teuthology.orchestra.run.smithi080:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:21:35.125 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:21:35.216 DEBUG:teuthology.orchestra.run.smithi080:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:21:35.249 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-04T15:21:35.250 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T15:21:35.317 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:21:35.368 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:21:35.410 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:21:35.506 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:21:35.540 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-04T15:21:35.540 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T15:21:35.609 DEBUG:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2024-03-04T15:21:35.611 DEBUG:teuthology.orchestra.run.smithi080:> sudo service rsyslog restart 2024-03-04T15:21:35.613 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2024-03-04T15:21:35.660 INFO:teuthology.orchestra.run.smithi080.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:21:35.670 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:21:35.712 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:21:36.245 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-04T15:21:36.253 INFO:teuthology.task.internal:Starting timer... 2024-03-04T15:21:36.253 INFO:teuthology.run_tasks:Running task pcp... 2024-03-04T15:21:36.263 INFO:teuthology.run_tasks:Running task selinux... 2024-03-04T15:21:36.273 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-04T15:21:36.274 DEBUG:teuthology.orchestra.run.smithi005:> sudo service auditd rotate 2024-03-04T15:21:36.351 INFO:teuthology.orchestra.run.smithi005.stdout:Rotating logs: 2024-03-04T15:21:36.352 DEBUG:teuthology.orchestra.run.smithi080:> sudo service auditd rotate 2024-03-04T15:21:36.402 INFO:teuthology.orchestra.run.smithi080.stdout:Rotating logs: 2024-03-04T15:21:36.402 DEBUG:teuthology.orchestra.run.smithi087:> sudo service auditd rotate 2024-03-04T15:21:36.464 INFO:teuthology.orchestra.run.smithi087.stdout:Rotating logs: 2024-03-04T15:21:36.465 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-04T15:21:36.465 DEBUG:teuthology.orchestra.run.smithi005:> /usr/sbin/getenforce 2024-03-04T15:21:36.495 INFO:teuthology.orchestra.run.smithi005.stdout:Permissive 2024-03-04T15:21:36.495 DEBUG:teuthology.orchestra.run.smithi080:> /usr/sbin/getenforce 2024-03-04T15:21:36.514 INFO:teuthology.orchestra.run.smithi080.stdout:Permissive 2024-03-04T15:21:36.514 DEBUG:teuthology.orchestra.run.smithi087:> /usr/sbin/getenforce 2024-03-04T15:21:36.538 INFO:teuthology.orchestra.run.smithi087.stdout:Permissive 2024-03-04T15:21:36.539 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi005.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi080.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi087.front.sepia.ceph.com': 'permissive'} 2024-03-04T15:21:36.539 DEBUG:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:21:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:21:36.598 DEBUG:teuthology.orchestra.run.smithi080:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:21:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:21:36.626 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:21:36.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:21:36.657 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-04T15:21:36.657 DEBUG:teuthology.orchestra.run.smithi005:> sudo /usr/sbin/setenforce permissive 2024-03-04T15:21:36.697 DEBUG:teuthology.orchestra.run.smithi080:> sudo /usr/sbin/setenforce permissive 2024-03-04T15:21:36.732 DEBUG:teuthology.orchestra.run.smithi087:> sudo /usr/sbin/setenforce permissive 2024-03-04T15:21:36.774 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-04T15:21:36.784 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-04T15:21:36.814 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-04T15:21:36.815 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi005.front.sepia.ceph.com,smithi080.front.sepia.ceph.com,smithi087.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-04T15:26:29.007 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi005.front.sepia.ceph.com'), Remote(name='ubuntu@smithi080.front.sepia.ceph.com'), Remote(name='ubuntu@smithi087.front.sepia.ceph.com')] 2024-03-04T15:26:29.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-04T15:26:29.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:26:29.108 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-03-04T15:26:29.190 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-04T15:26:29.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-04T15:26:29.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:26:29.312 DEBUG:teuthology.orchestra.run.smithi080:> true 2024-03-04T15:26:29.396 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-04T15:26:29.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-03-04T15:26:29.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:26:29.494 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-03-04T15:26:29.584 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-03-04T15:26:29.588 INFO:teuthology.run_tasks:Running task clock... 2024-03-04T15:26:29.600 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-04T15:26:29.600 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T15:26:29.601 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:26:29.603 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T15:26:29.603 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:26:29.605 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T15:26:29.605 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:26:29.636 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T15:26:29.637 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T15:26:29.654 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T15:26:29.654 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T15:26:29.656 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T15:26:29.675 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T15:26:29.684 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: ntpd: command not found 2024-03-04T15:26:29.685 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: ntpd: command not found 2024-03-04T15:26:29.698 INFO:teuthology.orchestra.run.smithi080.stdout:506 Cannot talk to daemon 2024-03-04T15:26:29.699 INFO:teuthology.orchestra.run.smithi005.stdout:506 Cannot talk to daemon 2024-03-04T15:26:29.704 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: ntpd: command not found 2024-03-04T15:26:29.714 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T15:26:29.716 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T15:26:29.718 INFO:teuthology.orchestra.run.smithi087.stdout:506 Cannot talk to daemon 2024-03-04T15:26:29.730 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T15:26:29.733 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T15:26:29.735 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T15:26:29.751 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T15:26:29.785 INFO:teuthology.orchestra.run.smithi080.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:26:29.785 INFO:teuthology.orchestra.run.smithi005.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:26:29.789 INFO:teuthology.orchestra.run.smithi080.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:26:29.789 INFO:teuthology.orchestra.run.smithi080.stdout:=============================================================================== 2024-03-04T15:26:29.789 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:26:29.789 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:26:29.789 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:26:29.789 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:26:29.790 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:26:29.790 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2024-03-04T15:26:29.790 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:26:29.790 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:26:29.790 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:26:29.790 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:26:29.802 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:26:29.806 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:26:29.806 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-03-04T15:26:29.806 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:26:29.806 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:26:29.806 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:26:29.807 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:26:29.807 INFO:teuthology.run_tasks:Running task pexec... 2024-03-04T15:26:29.816 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-04T15:26:29.816 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi005.front.sepia.ceph.com 2024-03-04T15:26:29.817 DEBUG:teuthology.orchestra.run.smithi005:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-04T15:26:29.817 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi080.front.sepia.ceph.com 2024-03-04T15:26:29.817 DEBUG:teuthology.orchestra.run.smithi080:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-04T15:26:29.817 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi087.front.sepia.ceph.com 2024-03-04T15:26:29.817 DEBUG:teuthology.orchestra.run.smithi087:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-04T15:26:30.408 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:00:53 ago on Mon 04 Mar 2024 03:25:37 PM UTC. 2024-03-04T15:26:30.422 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:51 ago on Mon 04 Mar 2024 03:25:39 PM UTC. 2024-03-04T15:26:30.427 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:52 ago on Mon 04 Mar 2024 03:25:38 PM UTC. 2024-03-04T15:26:30.569 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-03-04T15:26:30.589 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-03-04T15:26:30.590 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-04T15:26:30.590 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-03-04T15:26:30.590 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-04T15:26:30.591 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-03-04T15:26:30.591 INFO:teuthology.orchestra.run.smithi080.stdout: runc x86_64 4:1.1.12-2.el9 CentOS-AppStream 3.1 M 2024-03-04T15:26:30.591 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-04T15:26:30.591 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-03-04T15:26:30.591 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-03-04T15:26:30.591 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-03-04T15:26:30.591 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-04T15:26:30.591 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 3.1 M 2024-03-04T15:26:30.591 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 9.8 M 2024-03-04T15:26:30.591 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-03-04T15:26:30.592 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-04T15:26:30.596 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-04T15:26:30.596 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-03-04T15:26:30.596 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-04T15:26:30.596 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-04T15:26:30.596 INFO:teuthology.orchestra.run.smithi005.stdout: runc x86_64 4:1.1.12-2.el9 CentOS-AppStream 3.1 M 2024-03-04T15:26:30.596 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:26:30.597 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-04T15:26:30.597 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-04T15:26:30.597 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-03-04T15:26:30.597 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:26:30.597 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 3.1 M 2024-03-04T15:26:30.597 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 9.8 M 2024-03-04T15:26:30.597 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-04T15:26:30.605 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-04T15:26:30.605 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-03-04T15:26:30.605 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-04T15:26:30.605 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-03-04T15:26:30.605 INFO:teuthology.orchestra.run.smithi087.stdout: runc x86_64 4:1.1.12-2.el9 CentOS-AppStream 3.1 M 2024-03-04T15:26:30.605 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-04T15:26:30.605 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-03-04T15:26:30.605 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-03-04T15:26:30.605 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-03-04T15:26:30.606 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-04T15:26:30.606 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.1 M 2024-03-04T15:26:30.606 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 9.8 M 2024-03-04T15:26:30.606 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-03-04T15:26:31.270 INFO:teuthology.orchestra.run.smithi080.stdout:runc-1.1.12-2.el9.x86_64.rpm 4.6 MB/s | 3.1 MB 00:00 2024-03-04T15:26:31.271 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:26:31.271 INFO:teuthology.orchestra.run.smithi080.stdout:Total 4.6 MB/s | 3.1 MB 00:00 2024-03-04T15:26:31.271 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-03-04T15:26:31.274 INFO:teuthology.orchestra.run.smithi005.stdout:runc-1.1.12-2.el9.x86_64.rpm 4.6 MB/s | 3.1 MB 00:00 2024-03-04T15:26:31.274 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:26:31.275 INFO:teuthology.orchestra.run.smithi005.stdout:Total 4.6 MB/s | 3.1 MB 00:00 2024-03-04T15:26:31.275 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-04T15:26:31.276 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-03-04T15:26:31.276 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-03-04T15:26:31.281 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-04T15:26:31.281 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-04T15:26:31.302 INFO:teuthology.orchestra.run.smithi087.stdout:runc-1.1.12-2.el9.x86_64.rpm 4.5 MB/s | 3.1 MB 00:00 2024-03-04T15:26:31.302 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:26:31.302 INFO:teuthology.orchestra.run.smithi087.stdout:Total 4.5 MB/s | 3.1 MB 00:00 2024-03-04T15:26:31.303 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-03-04T15:26:31.308 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-03-04T15:26:31.308 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-03-04T15:26:31.347 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-03-04T15:26:31.347 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-03-04T15:26:31.358 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-04T15:26:31.358 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-04T15:26:31.379 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-03-04T15:26:31.379 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-03-04T15:26:31.554 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-03-04T15:26:31.575 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-03-04T15:26:31.579 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-04T15:26:31.648 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:26:31.680 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:26:31.680 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:26:32.189 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:26:32.238 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:26:32.244 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:26:32.518 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:26:32.518 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-04T15:26:32.519 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-03-04T15:26:32.519 INFO:teuthology.orchestra.run.smithi087.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-04T15:26:32.519 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-04T15:26:32.519 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-03-04T15:26:32.554 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:26:32.554 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:26:32.554 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-04T15:26:32.554 INFO:teuthology.orchestra.run.smithi005.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-04T15:26:32.554 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:26:32.554 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-04T15:26:32.600 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:26:32.600 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-04T15:26:32.600 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-03-04T15:26:32.600 INFO:teuthology.orchestra.run.smithi080.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-04T15:26:32.600 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-03-04T15:26:32.600 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-03-04T15:26:32.749 DEBUG:teuthology.parallel:result is None 2024-03-04T15:26:32.823 DEBUG:teuthology.parallel:result is None 2024-03-04T15:26:32.826 DEBUG:teuthology.parallel:result is None 2024-03-04T15:26:32.826 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-04T15:26:32.839 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-04T15:26:32.839 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-04T15:26:32.840 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:26:32.854 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T15:26:32.854 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_1 2024-03-04T15:26:32.908 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T15:26:32.909 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:26:32.909 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-04T15:26:32.909 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:26:32.909 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:26:32.909 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-04 15:26:27.936206673 +0000 2024-03-04T15:26:32.909 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-04 15:26:04.828636430 +0000 2024-03-04T15:26:32.909 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-04 15:26:04.828636430 +0000 2024-03-04T15:26:32.909 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-04 15:26:04.828636430 +0000 2024-03-04T15:26:32.909 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T15:26:32.974 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-04T15:26:32.974 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-04T15:26:32.974 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 8.9895e-05 s, 5.7 MB/s 2024-03-04T15:26:32.975 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T15:26:33.030 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_2 2024-03-04T15:26:33.084 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T15:26:33.085 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:26:33.085 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-04T15:26:33.085 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:26:33.085 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:26:33.085 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-04 15:26:27.936206673 +0000 2024-03-04T15:26:33.085 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-04 15:26:05.159630274 +0000 2024-03-04T15:26:33.085 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-04 15:26:05.159630274 +0000 2024-03-04T15:26:33.085 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-04 15:26:05.159630274 +0000 2024-03-04T15:26:33.085 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T15:26:33.149 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-04T15:26:33.149 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-04T15:26:33.150 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000101778 s, 5.0 MB/s 2024-03-04T15:26:33.151 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T15:26:33.206 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_3 2024-03-04T15:26:33.261 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T15:26:33.261 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:26:33.261 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-04T15:26:33.261 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:26:33.261 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:26:33.261 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-04 15:26:27.936206673 +0000 2024-03-04T15:26:33.261 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-04 15:26:05.486624192 +0000 2024-03-04T15:26:33.261 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-04 15:26:05.486624192 +0000 2024-03-04T15:26:33.261 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-04 15:26:05.486624192 +0000 2024-03-04T15:26:33.262 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T15:26:33.326 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-04T15:26:33.326 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-04T15:26:33.326 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 8.3448e-05 s, 6.1 MB/s 2024-03-04T15:26:33.327 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T15:26:33.383 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_4 2024-03-04T15:26:33.437 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T15:26:33.437 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:26:33.437 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-04T15:26:33.437 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:26:33.438 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:26:33.438 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-04 15:26:27.936206673 +0000 2024-03-04T15:26:33.438 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-04 15:26:05.998614670 +0000 2024-03-04T15:26:33.438 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-04 15:26:05.998614670 +0000 2024-03-04T15:26:33.438 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-04 15:26:05.998614670 +0000 2024-03-04T15:26:33.438 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T15:26:33.502 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-04T15:26:33.503 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-04T15:26:33.503 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 9.4979e-05 s, 5.4 MB/s 2024-03-04T15:26:33.504 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T15:26:33.559 DEBUG:teuthology.orchestra.run.smithi005:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-04T15:26:33.741 INFO:teuthology.orchestra.run.smithi005.stdout:loop 2024-03-04T15:26:33.743 INFO:tasks.nvme_loop:Connecting nvme_loop smithi005:/dev/vg_nvme/lv_1... 2024-03-04T15:26:33.743 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-04T15:26:33.780 INFO:teuthology.orchestra.run.smithi005.stdout:1 2024-03-04T15:26:33.817 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/vg_nvme/lv_11 2024-03-04T15:26:33.853 INFO:tasks.nvme_loop:Connecting nvme_loop smithi005:/dev/vg_nvme/lv_2... 2024-03-04T15:26:33.853 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-04T15:26:33.932 INFO:teuthology.orchestra.run.smithi005.stdout:1 2024-03-04T15:26:33.971 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/vg_nvme/lv_21 2024-03-04T15:26:34.012 INFO:tasks.nvme_loop:Connecting nvme_loop smithi005:/dev/vg_nvme/lv_3... 2024-03-04T15:26:34.013 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-04T15:26:34.089 INFO:teuthology.orchestra.run.smithi005.stdout:1 2024-03-04T15:26:34.128 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/vg_nvme/lv_31 2024-03-04T15:26:34.158 INFO:tasks.nvme_loop:Connecting nvme_loop smithi005:/dev/vg_nvme/lv_4... 2024-03-04T15:26:34.158 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-04T15:26:34.237 INFO:teuthology.orchestra.run.smithi005.stdout:1 2024-03-04T15:26:34.276 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/vg_nvme/lv_41 2024-03-04T15:26:34.305 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-04T15:26:34.305 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:26:34.360 DEBUG:teuthology.orchestra.run.smithi005:> sudo nvme list 2024-03-04T15:26:34.426 INFO:teuthology.orchestra.run.smithi005.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-04T15:26:34.427 INFO:teuthology.orchestra.run.smithi005.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-04T15:26:34.427 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/nvme0n1 PHMB751300EW480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-03-04T15:26:34.427 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/nvme1n1 1e119cb573918dece0e3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:26:34.427 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/nvme2n1 c9964ea2e74c4abd4603 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:26:34.427 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/nvme3n1 7f5456628a5167fe05c3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:26:34.427 INFO:teuthology.orchestra.run.smithi005.stdout:/dev/nvme4n1 5935cc579deaf4a58ac6 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:26:34.428 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-04T15:26:34.428 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-04T15:26:34.428 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/scratch_devs 2024-03-04T15:26:34.494 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-04T15:26:34.494 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:26:34.512 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T15:26:34.512 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_1 2024-03-04T15:26:34.568 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T15:26:34.569 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:26:34.569 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-04T15:26:34.569 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:26:34.569 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:26:34.569 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-04 15:26:27.924909808 +0000 2024-03-04T15:26:34.569 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-04 15:26:04.862391288 +0000 2024-03-04T15:26:34.569 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-04 15:26:04.862391288 +0000 2024-03-04T15:26:34.569 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-04 15:26:04.862391288 +0000 2024-03-04T15:26:34.569 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T15:26:34.637 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-04T15:26:34.637 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-04T15:26:34.638 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000272413 s, 1.9 MB/s 2024-03-04T15:26:34.639 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T15:26:34.696 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_2 2024-03-04T15:26:34.753 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T15:26:34.753 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:26:34.753 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-04T15:26:34.753 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:26:34.753 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:26:34.753 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-04 15:26:27.924909808 +0000 2024-03-04T15:26:34.753 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-04 15:26:05.225383710 +0000 2024-03-04T15:26:34.753 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-04 15:26:05.225383710 +0000 2024-03-04T15:26:34.753 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-04 15:26:05.225383710 +0000 2024-03-04T15:26:34.754 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T15:26:34.820 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-04T15:26:34.820 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-04T15:26:34.820 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000213017 s, 2.4 MB/s 2024-03-04T15:26:34.821 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T15:26:34.877 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_3 2024-03-04T15:26:34.933 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T15:26:34.933 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:26:34.933 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-04T15:26:34.933 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:26:34.933 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:26:34.933 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-04 15:26:27.924909808 +0000 2024-03-04T15:26:34.934 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-04 15:26:05.563376654 +0000 2024-03-04T15:26:34.934 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-04 15:26:05.563376654 +0000 2024-03-04T15:26:34.934 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-04 15:26:05.563376654 +0000 2024-03-04T15:26:34.934 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T15:26:34.998 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-04T15:26:34.998 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-04T15:26:34.998 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000223947 s, 2.3 MB/s 2024-03-04T15:26:34.999 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T15:26:35.055 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_4 2024-03-04T15:26:35.111 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T15:26:35.111 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:26:35.111 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-04T15:26:35.112 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:26:35.112 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:26:35.112 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-04 15:26:27.924909808 +0000 2024-03-04T15:26:35.112 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-04 15:26:05.923369138 +0000 2024-03-04T15:26:35.112 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-04 15:26:05.923369138 +0000 2024-03-04T15:26:35.112 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-04 15:26:05.923369138 +0000 2024-03-04T15:26:35.113 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T15:26:35.176 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-04T15:26:35.176 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-04T15:26:35.176 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000169958 s, 3.0 MB/s 2024-03-04T15:26:35.177 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T15:26:35.232 DEBUG:teuthology.orchestra.run.smithi080:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-04T15:26:35.373 INFO:teuthology.orchestra.run.smithi080.stdout:loop 2024-03-04T15:26:35.375 INFO:tasks.nvme_loop:Connecting nvme_loop smithi080:/dev/vg_nvme/lv_1... 2024-03-04T15:26:35.375 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-04T15:26:35.413 INFO:teuthology.orchestra.run.smithi080.stdout:1 2024-03-04T15:26:35.450 INFO:teuthology.orchestra.run.smithi080.stdout:/dev/vg_nvme/lv_11 2024-03-04T15:26:35.479 INFO:tasks.nvme_loop:Connecting nvme_loop smithi080:/dev/vg_nvme/lv_2... 2024-03-04T15:26:35.479 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-04T15:26:35.558 INFO:teuthology.orchestra.run.smithi080.stdout:1 2024-03-04T15:26:35.595 INFO:teuthology.orchestra.run.smithi080.stdout:/dev/vg_nvme/lv_21 2024-03-04T15:26:35.623 INFO:tasks.nvme_loop:Connecting nvme_loop smithi080:/dev/vg_nvme/lv_3... 2024-03-04T15:26:35.623 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-04T15:26:35.703 INFO:teuthology.orchestra.run.smithi080.stdout:1 2024-03-04T15:26:35.743 INFO:teuthology.orchestra.run.smithi080.stdout:/dev/vg_nvme/lv_31 2024-03-04T15:26:35.779 INFO:tasks.nvme_loop:Connecting nvme_loop smithi080:/dev/vg_nvme/lv_4... 2024-03-04T15:26:35.779 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-04T15:26:35.857 INFO:teuthology.orchestra.run.smithi080.stdout:1 2024-03-04T15:26:35.895 INFO:teuthology.orchestra.run.smithi080.stdout:/dev/vg_nvme/lv_41 2024-03-04T15:26:35.926 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-04T15:26:35.926 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:26:35.981 DEBUG:teuthology.orchestra.run.smithi080:> sudo nvme list 2024-03-04T15:26:36.048 INFO:teuthology.orchestra.run.smithi080.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-04T15:26:36.049 INFO:teuthology.orchestra.run.smithi080.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-04T15:26:36.049 INFO:teuthology.orchestra.run.smithi080.stdout:/dev/nvme0n1 PHFT6204014T400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-04T15:26:36.049 INFO:teuthology.orchestra.run.smithi080.stdout:/dev/nvme1n1 cace3921c89ec99838a6 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:26:36.049 INFO:teuthology.orchestra.run.smithi080.stdout:/dev/nvme2n1 8ad86045fb201c52a8cf Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:26:36.049 INFO:teuthology.orchestra.run.smithi080.stdout:/dev/nvme3n1 b80fa812465f5ff56a52 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:26:36.050 INFO:teuthology.orchestra.run.smithi080.stdout:/dev/nvme4n1 1f4db6893dd4e45e92cc Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:26:36.051 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-04T15:26:36.051 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-04T15:26:36.051 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/scratch_devs 2024-03-04T15:26:36.119 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-04T15:26:36.120 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:26:36.139 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T15:26:36.139 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_1 2024-03-04T15:26:36.198 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T15:26:36.198 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:26:36.199 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-04T15:26:36.199 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:26:36.199 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:26:36.199 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-03-04 15:26:27.943463898 +0000 2024-03-04T15:26:36.199 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-03-04 15:26:05.160951004 +0000 2024-03-04T15:26:36.199 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-03-04 15:26:05.160951004 +0000 2024-03-04T15:26:36.199 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-03-04 15:26:05.160951004 +0000 2024-03-04T15:26:36.200 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T15:26:36.269 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-03-04T15:26:36.269 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-03-04T15:26:36.269 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000221832 s, 2.3 MB/s 2024-03-04T15:26:36.270 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T15:26:36.330 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_2 2024-03-04T15:26:36.388 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T15:26:36.388 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:26:36.388 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-04T15:26:36.388 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:26:36.388 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:26:36.388 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-03-04 15:26:27.943463898 +0000 2024-03-04T15:26:36.388 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-03-04 15:26:05.504943649 +0000 2024-03-04T15:26:36.389 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-03-04 15:26:05.504943649 +0000 2024-03-04T15:26:36.389 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-03-04 15:26:05.504943649 +0000 2024-03-04T15:26:36.389 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T15:26:36.457 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-03-04T15:26:36.457 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-03-04T15:26:36.457 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000177611 s, 2.9 MB/s 2024-03-04T15:26:36.458 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T15:26:36.516 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_3 2024-03-04T15:26:36.576 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T15:26:36.576 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:26:36.576 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-04T15:26:36.576 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:26:36.577 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:26:36.577 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-03-04 15:26:27.943463898 +0000 2024-03-04T15:26:36.577 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-03-04 15:26:05.870935824 +0000 2024-03-04T15:26:36.577 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-03-04 15:26:05.870935824 +0000 2024-03-04T15:26:36.577 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-03-04 15:26:05.870935824 +0000 2024-03-04T15:26:36.577 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T15:26:36.645 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-03-04T15:26:36.645 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-03-04T15:26:36.645 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000217909 s, 2.3 MB/s 2024-03-04T15:26:36.646 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T15:26:36.704 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_4 2024-03-04T15:26:36.763 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T15:26:36.763 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:26:36.764 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-04T15:26:36.764 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:26:36.764 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:26:36.764 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-03-04 15:26:27.943463898 +0000 2024-03-04T15:26:36.764 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-03-04 15:26:06.216928426 +0000 2024-03-04T15:26:36.764 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-03-04 15:26:06.216928426 +0000 2024-03-04T15:26:36.764 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-03-04 15:26:06.216928426 +0000 2024-03-04T15:26:36.764 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T15:26:36.831 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-03-04T15:26:36.831 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-03-04T15:26:36.831 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000175441 s, 2.9 MB/s 2024-03-04T15:26:36.832 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T15:26:36.889 DEBUG:teuthology.orchestra.run.smithi087:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-04T15:26:37.062 INFO:teuthology.orchestra.run.smithi087.stdout:loop 2024-03-04T15:26:37.064 INFO:tasks.nvme_loop:Connecting nvme_loop smithi087:/dev/vg_nvme/lv_1... 2024-03-04T15:26:37.064 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-04T15:26:37.101 INFO:teuthology.orchestra.run.smithi087.stdout:1 2024-03-04T15:26:37.139 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/vg_nvme/lv_11 2024-03-04T15:26:37.179 INFO:tasks.nvme_loop:Connecting nvme_loop smithi087:/dev/vg_nvme/lv_2... 2024-03-04T15:26:37.179 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-04T15:26:37.259 INFO:teuthology.orchestra.run.smithi087.stdout:1 2024-03-04T15:26:37.297 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/vg_nvme/lv_21 2024-03-04T15:26:37.334 INFO:tasks.nvme_loop:Connecting nvme_loop smithi087:/dev/vg_nvme/lv_3... 2024-03-04T15:26:37.334 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-04T15:26:37.413 INFO:teuthology.orchestra.run.smithi087.stdout:1 2024-03-04T15:26:37.452 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/vg_nvme/lv_31 2024-03-04T15:26:37.489 INFO:tasks.nvme_loop:Connecting nvme_loop smithi087:/dev/vg_nvme/lv_4... 2024-03-04T15:26:37.489 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-04T15:26:37.568 INFO:teuthology.orchestra.run.smithi087.stdout:1 2024-03-04T15:26:37.610 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/vg_nvme/lv_41 2024-03-04T15:26:37.640 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-04T15:26:37.640 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:26:37.700 DEBUG:teuthology.orchestra.run.smithi087:> sudo nvme list 2024-03-04T15:26:37.766 INFO:teuthology.orchestra.run.smithi087.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-04T15:26:37.766 INFO:teuthology.orchestra.run.smithi087.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-04T15:26:37.766 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/nvme0n1 PHFT620400NQ400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-04T15:26:37.766 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/nvme1n1 d88c439eac4861eb727a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:26:37.766 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/nvme2n1 95379ce2941eeb5812d0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:26:37.766 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/nvme3n1 7213ecafec5bce822b75 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:26:37.766 INFO:teuthology.orchestra.run.smithi087.stdout:/dev/nvme4n1 787da765e0afec5d3f29 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:26:37.767 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-04T15:26:37.767 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-04T15:26:37.767 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/scratch_devs 2024-03-04T15:26:37.835 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-04T15:26:37.932 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'global': {'mon election default strategy': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282'} 2024-03-04T15:26:37.933 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:26:37.933 INFO:tasks.cephadm:Cluster fsid is 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:26:37.933 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-04T15:26:37.933 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.5', 'mon.b': '172.21.15.80', 'mon.c': '172.21.15.87'} 2024-03-04T15:26:37.934 INFO:tasks.cephadm:First mon is mon.a on smithi005 2024-03-04T15:26:37.934 INFO:tasks.cephadm:First mgr is y 2024-03-04T15:26:37.934 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-04T15:26:37.934 DEBUG:teuthology.orchestra.run.smithi005:> sudo hostname $(hostname -s) 2024-03-04T15:26:37.966 DEBUG:teuthology.orchestra.run.smithi080:> sudo hostname $(hostname -s) 2024-03-04T15:26:37.999 DEBUG:teuthology.orchestra.run.smithi087:> sudo hostname $(hostname -s) 2024-03-04T15:26:38.026 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-04T15:26:38.026 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:26:38.211 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77294/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1826-g9ca0b5de', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1826.g9ca0b5de'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-04 12:37:35.725599', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-asm0deuz-testing-2024-03-04-1211', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-04T15:26:38.373 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-asm0deuz-testing-2024-03-04-1211, sha1 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:26:38.375 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/x86_64/flavors/default/cephadm 2024-03-04T15:26:38.375 DEBUG:teuthology.orchestra.run.smithi005:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T15:26:39.398 INFO:teuthology.orchestra.run.smithi005.stdout:-rw-r--r--. 1 ubuntu ubuntu 767364 Mar 4 15:26 /home/ubuntu/cephtest/cephadm 2024-03-04T15:26:39.398 DEBUG:teuthology.orchestra.run.smithi080:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T15:26:40.230 INFO:teuthology.orchestra.run.smithi080.stdout:-rw-r--r--. 1 ubuntu ubuntu 767364 Mar 4 15:26 /home/ubuntu/cephtest/cephadm 2024-03-04T15:26:40.230 DEBUG:teuthology.orchestra.run.smithi087:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T15:26:41.988 INFO:teuthology.orchestra.run.smithi087.stdout:-rw-r--r--. 1 ubuntu ubuntu 767364 Mar 4 15:26 /home/ubuntu/cephtest/cephadm 2024-03-04T15:26:41.988 DEBUG:teuthology.orchestra.run.smithi005:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T15:26:42.007 DEBUG:teuthology.orchestra.run.smithi080:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T15:26:42.026 DEBUG:teuthology.orchestra.run.smithi087:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T15:26:42.057 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 on all hosts... 2024-03-04T15:26:42.057 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 pull 2024-03-04T15:26:42.059 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 pull 2024-03-04T15:26:42.068 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 pull 2024-03-04T15:26:42.858 INFO:teuthology.orchestra.run.smithi080.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282... 2024-03-04T15:26:42.862 INFO:teuthology.orchestra.run.smithi005.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282... 2024-03-04T15:26:42.937 INFO:teuthology.orchestra.run.smithi087.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282... 2024-03-04T15:30:05.827 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-03-04T15:30:05.827 INFO:teuthology.orchestra.run.smithi087.stdout: "ceph_version": "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)", 2024-03-04T15:30:05.827 INFO:teuthology.orchestra.run.smithi087.stdout: "image_id": "0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692", 2024-03-04T15:30:05.827 INFO:teuthology.orchestra.run.smithi087.stdout: "repo_digests": [ 2024-03-04T15:30:05.827 INFO:teuthology.orchestra.run.smithi087.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977" 2024-03-04T15:30:05.827 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-03-04T15:30:05.827 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-03-04T15:35:48.834 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-04T15:35:48.835 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph_version": "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)", 2024-03-04T15:35:48.835 INFO:teuthology.orchestra.run.smithi005.stdout: "image_id": "0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692", 2024-03-04T15:35:48.835 INFO:teuthology.orchestra.run.smithi005.stdout: "repo_digests": [ 2024-03-04T15:35:48.835 INFO:teuthology.orchestra.run.smithi005.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977" 2024-03-04T15:35:48.835 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-03-04T15:35:48.835 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-04T15:36:53.240 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-03-04T15:36:53.240 INFO:teuthology.orchestra.run.smithi080.stdout: "ceph_version": "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)", 2024-03-04T15:36:53.240 INFO:teuthology.orchestra.run.smithi080.stdout: "image_id": "0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692", 2024-03-04T15:36:53.240 INFO:teuthology.orchestra.run.smithi080.stdout: "repo_digests": [ 2024-03-04T15:36:53.240 INFO:teuthology.orchestra.run.smithi080.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977" 2024-03-04T15:36:53.240 INFO:teuthology.orchestra.run.smithi080.stdout: ] 2024-03-04T15:36:53.240 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-03-04T15:36:53.267 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph 2024-03-04T15:36:53.303 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /etc/ceph 2024-03-04T15:36:53.334 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph 2024-03-04T15:36:53.385 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 777 /etc/ceph 2024-03-04T15:36:53.415 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 777 /etc/ceph 2024-03-04T15:36:53.442 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 777 /etc/ceph 2024-03-04T15:36:53.473 INFO:tasks.cephadm:Writing seed config... 2024-03-04T15:36:53.474 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-04T15:36:53.474 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-04T15:36:53.474 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-03-04T15:36:53.474 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-04T15:36:53.475 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-04T15:36:53.475 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-04T15:36:53.475 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-04T15:36:53.475 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-04T15:36:53.475 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-04T15:36:53.475 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-04T15:36:53.476 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-04T15:36:53.476 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-04T15:36:53.492 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 97c626fc-da3b-11ee-95c6-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-04T15:36:53.493 DEBUG:teuthology.orchestra.run.smithi005:mon.a> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mon.a.service 2024-03-04T15:36:53.535 DEBUG:teuthology.orchestra.run.smithi005:mgr.y> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mgr.y.service 2024-03-04T15:36:53.577 INFO:tasks.cephadm:Bootstrapping... 2024-03-04T15:36:53.577 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -v bootstrap --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.5 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:36:53.774 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:36:53.775 INFO:teuthology.orchestra.run.smithi005.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282', '-v', 'bootstrap', '--fsid', '97c626fc-da3b-11ee-95c6-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.5', '--skip-admin-label'] 2024-03-04T15:36:53.800 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T15:36:53.801 INFO:teuthology.orchestra.run.smithi005.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-04T15:36:53.801 INFO:teuthology.orchestra.run.smithi005.stdout:Verifying podman|docker is present... 2024-03-04T15:36:53.824 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T15:36:53.824 INFO:teuthology.orchestra.run.smithi005.stdout:Verifying lvm2 is present... 2024-03-04T15:36:53.824 INFO:teuthology.orchestra.run.smithi005.stdout:Verifying time synchronization is in place... 2024-03-04T15:36:53.832 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-04T15:36:53.832 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-04T15:36:53.837 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-04T15:36:53.837 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout inactive 2024-03-04T15:36:53.843 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout enabled 2024-03-04T15:36:53.849 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout active 2024-03-04T15:36:53.849 INFO:teuthology.orchestra.run.smithi005.stdout:Unit chronyd.service is enabled and running 2024-03-04T15:36:53.849 INFO:teuthology.orchestra.run.smithi005.stdout:Repeating the final host check... 2024-03-04T15:36:53.873 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T15:36:53.873 INFO:teuthology.orchestra.run.smithi005.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-04T15:36:53.873 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl is present 2024-03-04T15:36:53.873 INFO:teuthology.orchestra.run.smithi005.stdout:lvcreate is present 2024-03-04T15:36:53.879 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-04T15:36:53.880 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-04T15:36:53.885 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-04T15:36:53.885 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout inactive 2024-03-04T15:36:53.891 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout enabled 2024-03-04T15:36:53.897 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout active 2024-03-04T15:36:53.897 INFO:teuthology.orchestra.run.smithi005.stdout:Unit chronyd.service is enabled and running 2024-03-04T15:36:53.897 INFO:teuthology.orchestra.run.smithi005.stdout:Host looks OK 2024-03-04T15:36:53.897 INFO:teuthology.orchestra.run.smithi005.stdout:Cluster fsid: 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:36:53.897 INFO:teuthology.orchestra.run.smithi005.stdout:Acquiring lock 140551764796800 on /run/cephadm/97c626fc-da3b-11ee-95c6-87774f69a715.lock 2024-03-04T15:36:53.898 INFO:teuthology.orchestra.run.smithi005.stdout:Lock 140551764796800 acquired on /run/cephadm/97c626fc-da3b-11ee-95c6-87774f69a715.lock 2024-03-04T15:36:53.898 INFO:teuthology.orchestra.run.smithi005.stdout:Verifying IP 172.21.15.5 port 3300 ... 2024-03-04T15:36:53.898 INFO:teuthology.orchestra.run.smithi005.stdout:Verifying IP 172.21.15.5 port 6789 ... 2024-03-04T15:36:53.898 INFO:teuthology.orchestra.run.smithi005.stdout:Base mon IP(s) is [172.21.15.5:3300, 172.21.15.5:6789], mon addrv is [v2:172.21.15.5:3300,v1:172.21.15.5:6789] 2024-03-04T15:36:53.901 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.5 metric 100 2024-03-04T15:36:53.901 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.5 metric 100 2024-03-04T15:36:53.903 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-04T15:36:53.904 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-04T15:36:53.904 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-04T15:36:53.906 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-04T15:36:53.906 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-04T15:36:53.906 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-04T15:36:53.906 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-04T15:36:53.906 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1442/64 scope link noprefixroute 2024-03-04T15:36:53.906 INFO:teuthology.orchestra.run.smithi005.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-04T15:36:53.907 INFO:teuthology.orchestra.run.smithi005.stdout:Mon IP `172.21.15.5` is in CIDR network `172.21.0.0/20` 2024-03-04T15:36:53.907 INFO:teuthology.orchestra.run.smithi005.stdout:Mon IP `172.21.15.5` is in CIDR network `172.21.0.0/20` 2024-03-04T15:36:53.907 INFO:teuthology.orchestra.run.smithi005.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-04T15:36:53.907 INFO:teuthology.orchestra.run.smithi005.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-04T15:36:53.908 INFO:teuthology.orchestra.run.smithi005.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282... 2024-03-04T15:36:54.555 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stdout 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 2024-03-04T15:36:54.555 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282... 2024-03-04T15:36:54.555 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stderr Getting image source signatures 2024-03-04T15:36:54.555 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stderr Copying blob sha256:64fd3420507ff7dcf06d181c53a3dd949a37f619fe84875331c583b3b5f6f668 2024-03-04T15:36:54.555 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stderr Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-03-04T15:36:54.555 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stderr Copying config sha256:0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 2024-03-04T15:36:54.556 INFO:teuthology.orchestra.run.smithi005.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-04T15:36:55.518 INFO:teuthology.orchestra.run.smithi005.stdout:ceph: stdout ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev) 2024-03-04T15:36:55.518 INFO:teuthology.orchestra.run.smithi005.stdout:Ceph version: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev) 2024-03-04T15:36:55.518 INFO:teuthology.orchestra.run.smithi005.stdout:Extracting ceph user uid/gid from container image... 2024-03-04T15:36:56.581 INFO:teuthology.orchestra.run.smithi005.stdout:stat: stdout 167 167 2024-03-04T15:36:56.582 INFO:teuthology.orchestra.run.smithi005.stdout:Creating initial keys... 2024-03-04T15:36:57.412 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph-authtool: stdout AQCY6uVlE+E7OBAADcqpZQ26wSWc0HZCaM9SXA== 2024-03-04T15:36:58.459 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph-authtool: stdout AQCZ6uVlyh+RLRAAtlEByJxGDa9Bvg+gnlGn/w== 2024-03-04T15:36:59.440 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph-authtool: stdout AQCa6uVlbyRTLhAA8rAK1zei6UnjRMzox4KIXw== 2024-03-04T15:36:59.440 INFO:teuthology.orchestra.run.smithi005.stdout:Creating initial monmap... 2024-03-04T15:37:00.312 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-04T15:37:00.313 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-04T15:37:00.313 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:37:00.313 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-04T15:37:00.313 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool for a [v2:172.21.15.5:3300,v1:172.21.15.5:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-04T15:37:00.313 INFO:teuthology.orchestra.run.smithi005.stdout:setting min_mon_release = quincy 2024-03-04T15:37:00.313 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/monmaptool: set fsid to 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:37:00.313 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-04T15:37:00.313 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:37:00.313 INFO:teuthology.orchestra.run.smithi005.stdout:Creating mon... 2024-03-04T15:37:01.284 INFO:teuthology.orchestra.run.smithi005.stdout:create mon.a on 2024-03-04T15:37:01.623 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-04T15:37:01.798 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-97c626fc-da3b-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-97c626fc-da3b-11ee-95c6-87774f69a715.target. 2024-03-04T15:37:01.799 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-97c626fc-da3b-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-97c626fc-da3b-11ee-95c6-87774f69a715.target. 2024-03-04T15:37:02.038 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mon.a 2024-03-04T15:37:02.038 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Failed to reset failed state of unit ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mon.a.service: Unit ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mon.a.service not loaded. 2024-03-04T15:37:02.204 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-97c626fc-da3b-11ee-95c6-87774f69a715.target.wants/ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mon.a.service → /etc/systemd/system/ceph-97c626fc-da3b-11ee-95c6-87774f69a715@.service. 2024-03-04T15:37:02.699 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:37:02.699 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout disabled 2024-03-04T15:37:02.706 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:37:02.706 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout inactive 2024-03-04T15:37:02.706 INFO:teuthology.orchestra.run.smithi005.stdout:firewalld.service is not enabled 2024-03-04T15:37:02.706 INFO:teuthology.orchestra.run.smithi005.stdout:Not possible to enable service . firewalld.service is not available 2024-03-04T15:37:02.706 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for mon to start... 2024-03-04T15:37:02.707 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for mon... 2024-03-04T15:37:03.763 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout cluster: 2024-03-04T15:37:03.764 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout id: 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:37:03.764 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-04T15:37:03.764 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-03-04T15:37:03.764 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout services: 2024-03-04T15:37:03.764 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.517706s) 2024-03-04T15:37:03.764 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-04T15:37:03.764 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-04T15:37:03.764 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-03-04T15:37:03.764 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout data: 2024-03-04T15:37:03.764 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-04T15:37:03.764 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-04T15:37:03.764 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-04T15:37:03.765 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout pgs: 2024-03-04T15:37:03.765 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-03-04T15:37:03.765 INFO:teuthology.orchestra.run.smithi005.stdout:mon is available 2024-03-04T15:37:03.765 INFO:teuthology.orchestra.run.smithi005.stdout:Assimilating anything we can from ceph.conf... 2024-03-04T15:37:04.849 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:04 smithi005 ceph-mon[25370]: from='client.? 172.21.15.5:0/1646707742' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-04T15:37:04.852 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-03-04T15:37:04.852 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout [global] 2024-03-04T15:37:04.853 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout fsid = 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:37:04.853 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.5:3300,v1:172.21.15.5:6789] 2024-03-04T15:37:04.853 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-04T15:37:04.853 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-04T15:37:04.853 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-04T15:37:04.853 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-04T15:37:04.853 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-03-04T15:37:04.853 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-04T15:37:04.853 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-03-04T15:37:04.853 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-04T15:37:04.853 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-03-04T15:37:04.853 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout [osd] 2024-03-04T15:37:04.853 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-04T15:37:04.853 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-04T15:37:04.854 INFO:teuthology.orchestra.run.smithi005.stdout:Generating new minimal ceph.conf... 2024-03-04T15:37:05.866 INFO:teuthology.orchestra.run.smithi005.stdout:Restarting the monitor... 2024-03-04T15:37:05.930 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:05 smithi005 ceph-mon[25370]: from='client.? 172.21.15.5:0/4103049818' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:37:05.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:05 smithi005 systemd[1]: Stopping Ceph mon.a for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:37:06.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:06 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-a[25347]: 2024-03-04T15:37:06.126+0000 7ff34a324700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:37:06.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:06 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-a[25347]: 2024-03-04T15:37:06.126+0000 7ff34a324700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-04T15:37:06.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:06 smithi005 podman[25701]: 2024-03-04 15:37:06.239241608 +0000 UTC m=+0.212605017 container died 5579ebc357683a37dcff37e9068d22c05475f98f6e845b04950182ef6299a2f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-a, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-04T15:37:06.831 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:06 smithi005 podman[25701]: 2024-03-04 15:37:06.57355856 +0000 UTC m=+0.546921970 container cleanup 5579ebc357683a37dcff37e9068d22c05475f98f6e845b04950182ef6299a2f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T15:37:06.831 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:06 smithi005 bash[25701]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-a 2024-03-04T15:37:06.831 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:06 smithi005 podman[25720]: 2024-03-04 15:37:06.741037571 +0000 UTC m=+0.499608468 container remove 5579ebc357683a37dcff37e9068d22c05475f98f6e845b04950182ef6299a2f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-a, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T15:37:07.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:06 smithi005 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-04T15:37:07.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:06 smithi005 systemd[1]: Stopped Ceph mon.a for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:37:07.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:06 smithi005 systemd[1]: Starting Ceph mon.a for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:37:07.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 podman[25821]: 2024-03-04 15:37:07.062838112 +0000 UTC m=+0.018579862 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:37:07.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 podman[25821]: 2024-03-04 15:37:07.166816597 +0000 UTC m=+0.122558382 container create c077183e60f8a64973c94f9fb41f5cf34bc1c1088f3b71335aed0f8ec3a42d66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T15:37:07.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 podman[25821]: 2024-03-04 15:37:07.368914673 +0000 UTC m=+0.324656416 container init c077183e60f8a64973c94f9fb41f5cf34bc1c1088f3b71335aed0f8ec3a42d66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-a, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-04T15:37:07.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 podman[25821]: 2024-03-04 15:37:07.373874562 +0000 UTC m=+0.329616313 container start c077183e60f8a64973c94f9fb41f5cf34bc1c1088f3b71335aed0f8ec3a42d66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-a, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222) 2024-03-04T15:37:07.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T15:37:07.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev), process ceph-mon, pid 7 2024-03-04T15:37:07.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: pidfile_write: ignore empty --pid-file 2024-03-04T15:37:07.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: load: jerasure load: lrc 2024-03-04T15:37:07.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: RocksDB version: 7.9.2 2024-03-04T15:37:07.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Git sha 0 2024-03-04T15:37:07.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Compile date 2024-03-04 11:37:16 2024-03-04T15:37:07.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: DB SUMMARY 2024-03-04T15:37:07.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: DB Session ID: 472TQHAL6XYBS4CPTFYS 2024-03-04T15:37:07.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: CURRENT file: CURRENT 2024-03-04T15:37:07.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T15:37:07.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-04T15:37:07.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-04T15:37:07.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84601 ; 2024-03-04T15:37:07.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.error_if_exists: 0 2024-03-04T15:37:07.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.create_if_missing: 0 2024-03-04T15:37:07.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.paranoid_checks: 1 2024-03-04T15:37:07.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T15:37:07.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T15:37:07.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T15:37:07.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.env: 0x55a9804e4b20 2024-03-04T15:37:07.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T15:37:07.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.info_log: 0x55a9819778e0 2024-03-04T15:37:07.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T15:37:07.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.statistics: (nil) 2024-03-04T15:37:07.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.use_fsync: 0 2024-03-04T15:37:07.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_log_file_size: 0 2024-03-04T15:37:07.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T15:37:07.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T15:37:07.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T15:37:07.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T15:37:07.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.allow_fallocate: 1 2024-03-04T15:37:07.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T15:37:07.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T15:37:07.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.use_direct_reads: 0 2024-03-04T15:37:07.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T15:37:07.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T15:37:07.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.db_log_dir: 2024-03-04T15:37:07.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.wal_dir: 2024-03-04T15:37:07.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T15:37:07.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T15:37:07.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T15:37:07.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T15:37:07.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T15:37:07.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T15:37:07.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T15:37:07.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T15:37:07.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.write_buffer_manager: 0x55a98194b400 2024-03-04T15:37:07.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T15:37:07.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T15:37:07.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T15:37:07.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T15:37:07.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T15:37:07.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T15:37:07.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T15:37:07.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T15:37:07.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.unordered_write: 0 2024-03-04T15:37:07.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T15:37:07.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T15:37:07.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T15:37:07.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T15:37:07.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.row_cache: None 2024-03-04T15:37:07.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.wal_filter: None 2024-03-04T15:37:07.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T15:37:07.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T15:37:07.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.two_write_queues: 0 2024-03-04T15:37:07.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T15:37:07.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.wal_compression: 0 2024-03-04T15:37:07.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.atomic_flush: 0 2024-03-04T15:37:07.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T15:37:07.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T15:37:07.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T15:37:07.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.log_readahead_size: 0 2024-03-04T15:37:07.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T15:37:07.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T15:37:07.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T15:37:07.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T15:37:07.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T15:37:07.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T15:37:07.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T15:37:07.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_background_jobs: 2 2024-03-04T15:37:07.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_background_compactions: -1 2024-03-04T15:37:07.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_subcompactions: 1 2024-03-04T15:37:07.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T15:37:07.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T15:37:07.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T15:37:07.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T15:37:07.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T15:37:07.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T15:37:07.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T15:37:07.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T15:37:07.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_open_files: -1 2024-03-04T15:37:07.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T15:37:07.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T15:37:07.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T15:37:07.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T15:37:07.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_background_flushes: -1 2024-03-04T15:37:07.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Compression algorithms supported: 2024-03-04T15:37:07.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T15:37:07.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: kZSTD supported: 0 2024-03-04T15:37:07.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: kXpressCompression supported: 0 2024-03-04T15:37:07.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T15:37:07.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: kZlibCompression supported: 1 2024-03-04T15:37:07.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: kSnappyCompression supported: 1 2024-03-04T15:37:07.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: kLZ4Compression supported: 1 2024-03-04T15:37:07.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: kBZip2Compression supported: 0 2024-03-04T15:37:07.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T15:37:07.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T15:37:07.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-04T15:37:07.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T15:37:07.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T15:37:07.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.merge_operator: 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compaction_filter: None 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a9819779e0) 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: cache_index_and_filter_blocks: 1 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: pin_top_level_index_and_filter: 1 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: index_type: 0 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: data_block_index_type: 0 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: index_shortening: 1 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: checksum: 4 2024-03-04T15:37:07.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: no_block_cache: 0 2024-03-04T15:37:07.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache: 0x55a98197d090 2024-03-04T15:37:07.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache_name: BinnedLRUCache 2024-03-04T15:37:07.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache_options: 2024-03-04T15:37:07.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: capacity : 536870912 2024-03-04T15:37:07.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: num_shard_bits : 4 2024-03-04T15:37:07.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: strict_capacity_limit : 0 2024-03-04T15:37:07.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: high_pri_pool_ratio: 0.000 2024-03-04T15:37:07.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache_compressed: (nil) 2024-03-04T15:37:07.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: persistent_cache: (nil) 2024-03-04T15:37:07.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_size: 4096 2024-03-04T15:37:07.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_size_deviation: 10 2024-03-04T15:37:07.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_restart_interval: 16 2024-03-04T15:37:07.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: index_block_restart_interval: 1 2024-03-04T15:37:07.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: metadata_block_size: 4096 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: partition_filters: 0 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: use_delta_encoding: 1 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: filter_policy: bloomfilter 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: whole_key_filtering: 1 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: verify_compression: 0 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: read_amp_bytes_per_bit: 0 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: format_version: 5 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: enable_index_compression: 1 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_align: 0 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: max_auto_readahead_size: 262144 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: prepopulate_block_cache: 0 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: initial_auto_readahead_size: 8192 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T15:37:07.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compression: NoCompression 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.num_levels: 7 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:37:07.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:37:07.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T15:37:07.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T15:37:07.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T15:37:07.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T15:37:07.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:37:07.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:37:07.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T15:37:07.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T15:37:07.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:37:07.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T15:37:07.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T15:37:07.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T15:37:07.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T15:37:07.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T15:37:07.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T15:37:07.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T15:37:07.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T15:37:07.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T15:37:07.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T15:37:07.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T15:37:07.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T15:37:07.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T15:37:07.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T15:37:07.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T15:37:07.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T15:37:07.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T15:37:07.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T15:37:07.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T15:37:07.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T15:37:07.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.table_properties_collectors: 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.inplace_update_support: 0 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T15:37:07.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.bloom_locality: 0 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.max_successive_merges: 0 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.ttl: 2592000 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.enable_blob_files: false 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.min_blob_size: 0 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T15:37:07.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T15:37:07.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T15:37:07.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T15:37:07.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T15:37:07.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T15:37:07.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T15:37:07.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-04T15:37:07.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-04T15:37:07.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 21a77c21-4b72-441d-901e-a65415077ae5 2024-03-04T15:37:07.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709566627402596, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-04T15:37:07.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-04T15:37:07.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709566627403463, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81226, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79413, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73722, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709566627, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "21a77c21-4b72-441d-901e-a65415077ae5", "db_session_id": "472TQHAL6XYBS4CPTFYS", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-04T15:37:07.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709566627403548, "job": 1, "event": "recovery_finished"} 2024-03-04T15:37:07.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-04T15:37:07.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T15:37:07.450 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a981a70000 2024-03-04T15:37:07.450 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: DB pointer 0x55a981a5a000 2024-03-04T15:37:07.450 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T15:37:07.450 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T15:37:07.450 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** DB Stats ** 2024-03-04T15:37:07.450 INFO:journalctl@ceph.mon.a.smithi005.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:37:07.450 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T15:37:07.450 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:37:07.450 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:37:07.450 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T15:37:07.450 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:37:07.450 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:37:07.450 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-04T15:37:07.450 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** Compaction Stats [default] ** 2024-03-04T15:37:07.450 INFO:journalctl@ceph.mon.a.smithi005.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: L0 2/0 81.16 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: Sum 2/0 81.16 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** Compaction Stats [default] ** 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 134.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T15:37:07.451 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T15:37:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T15:37:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T15:37:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative compaction: 0.00 GB write, 27.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:37:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval compaction: 0.00 GB write, 27.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:37:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T15:37:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: Block cache BinnedLRUCache@0x55a98197d090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-04T15:37:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-04T15:37:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-04T15:37:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T15:37:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: starting mon.a rank 0 at public addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] at bind addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:37:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: mon.a@-1(???) e1 preinit fsid 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:37:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: mon.a@-1(???).mds e1 new map 2024-03-04T15:37:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: mon.a@-1(???).mds e1 print_map 2024-03-04T15:37:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: e1 2024-03-04T15:37:07.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T15:37:07.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T15:37:07.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: legacy client fscid: -1 2024-03-04T15:37:07.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-04T15:37:07.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: No filesystems configured 2024-03-04T15:37:07.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-04T15:37:07.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:37:07.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:37:07.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:37:07.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-04T15:37:07.457 INFO:teuthology.orchestra.run.smithi005.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-04T15:37:07.773 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 bash[25821]: c077183e60f8a64973c94f9fb41f5cf34bc1c1088f3b71335aed0f8ec3a42d66 2024-03-04T15:37:07.773 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 systemd[1]: Started Ceph mon.a for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:37:07.773 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-04T15:37:07.773 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: monmap e1: 1 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:37:07.773 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: fsmap 2024-03-04T15:37:07.774 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: osdmap e1: 0 total, 0 up, 0 in 2024-03-04T15:37:07.774 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:07 smithi005 ceph-mon[25858]: mgrmap e1: no daemons active 2024-03-04T15:37:08.496 INFO:teuthology.orchestra.run.smithi005.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-04T15:37:08.517 INFO:teuthology.orchestra.run.smithi005.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:37:08.518 INFO:teuthology.orchestra.run.smithi005.stdout:Creating mgr... 2024-03-04T15:37:08.518 INFO:teuthology.orchestra.run.smithi005.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-04T15:37:08.519 INFO:teuthology.orchestra.run.smithi005.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-04T15:37:08.765 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mgr.y 2024-03-04T15:37:08.765 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Failed to reset failed state of unit ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mgr.y.service: Unit ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mgr.y.service not loaded. 2024-03-04T15:37:08.915 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-97c626fc-da3b-11ee-95c6-87774f69a715.target.wants/ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-97c626fc-da3b-11ee-95c6-87774f69a715@.service. 2024-03-04T15:37:09.155 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:08 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/655874360' entity='client.admin' 2024-03-04T15:37:09.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:09 smithi005 podman[26094]: 2024-03-04 15:37:09.368670377 +0000 UTC m=+0.332091532 container start f139651d923a0a0e3d3e1b83b3e1f4540adfe0c7b76de056a8c8d10d6dc84da5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0) 2024-03-04T15:37:09.494 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:37:09.494 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout disabled 2024-03-04T15:37:09.502 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:37:09.502 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout inactive 2024-03-04T15:37:09.502 INFO:teuthology.orchestra.run.smithi005.stdout:firewalld.service is not enabled 2024-03-04T15:37:09.502 INFO:teuthology.orchestra.run.smithi005.stdout:Not possible to enable service . firewalld.service is not available 2024-03-04T15:37:09.510 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:37:09.510 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout disabled 2024-03-04T15:37:09.517 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:37:09.517 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout inactive 2024-03-04T15:37:09.517 INFO:teuthology.orchestra.run.smithi005.stdout:firewalld.service is not enabled 2024-03-04T15:37:09.517 INFO:teuthology.orchestra.run.smithi005.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-04T15:37:09.517 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for mgr to start... 2024-03-04T15:37:09.517 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for mgr... 2024-03-04T15:37:09.835 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:09 smithi005 bash[26094]: f139651d923a0a0e3d3e1b83b3e1f4540adfe0c7b76de056a8c8d10d6dc84da5 2024-03-04T15:37:09.835 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:09 smithi005 systemd[1]: Started Ceph mgr.y for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:37:10.150 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:09 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:09.934+0000 7f5a54a57200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:37:10.150 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:10 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:10.032+0000 7f5a54a57200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:37:10.430 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:10 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/1315025966' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:37:10.768 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-03-04T15:37:10.768 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:37:10.768 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "fsid": "97c626fc-da3b-11ee-95c6-87774f69a715", 2024-03-04T15:37:10.768 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:37:10.768 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:37:10.768 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:37:10.768 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:37:10.768 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:10.768 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:37:10.768 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:37:10.768 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:37:10.769 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:37:10.769 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:37:10.769 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T15:37:10.769 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:37:10.769 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-04T15:37:10.769 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:37:10.769 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:10.769 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T15:37:10.769 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:37:10.769 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:10.769 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:37:10.769 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:10.769 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:37:10.769 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:37:10.769 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:37:10.770 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:37:10.770 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:37:10.770 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:37:10.773 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:10.773 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:37:10.773 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:37:10.774 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:37:10.774 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:37:10.774 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:37:10.774 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:37:10.774 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:37:10.774 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:37:10.774 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:37:10.774 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:10.774 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:37:10.774 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:10.774 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:37:10.774 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:37:10.774 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:10.774 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:37:10.774 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:37:02.678368+0000", 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:37:10.775 INFO:teuthology.orchestra.run.smithi005.stdout:mgr not available, waiting (1/15)... 2024-03-04T15:37:11.062 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:10 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:10.725+0000 7f5a54a57200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:37:11.062 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:10 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:10.820+0000 7f5a54a57200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:37:11.063 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:10 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:10.963+0000 7f5a54a57200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:37:11.063 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:11 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:11.060+0000 7f5a54a57200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:37:11.414 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:11 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:11.144+0000 7f5a54a57200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:37:11.414 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:11 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:11.412+0000 7f5a54a57200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:37:11.680 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:11 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:11.523+0000 7f5a54a57200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:37:11.681 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:11 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:11.611+0000 7f5a54a57200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:37:12.180 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:11 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:11.707+0000 7f5a54a57200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:37:12.181 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:11 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:11.885+0000 7f5a54a57200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:37:13.181 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:12 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:12.891+0000 7f5a54a57200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:37:13.181 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:13 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:13.026+0000 7f5a54a57200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:37:13.473 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:13 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/853722377' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:37:13.473 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:13 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:13.375+0000 7f5a54a57200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:37:13.860 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:13 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:13.470+0000 7f5a54a57200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:37:13.860 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:13 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:13.671+0000 7f5a54a57200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:37:13.860 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:13 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:13.858+0000 7f5a54a57200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:37:13.990 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-03-04T15:37:13.990 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:37:13.990 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "fsid": "97c626fc-da3b-11ee-95c6-87774f69a715", 2024-03-04T15:37:13.990 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:37:13.990 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:37:13.990 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:37:13.990 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:37:13.990 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:13.991 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:37:13.992 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:37:13.992 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:37:13.992 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:37:13.992 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:37:13.992 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:37:13.995 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:13.995 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:37:13.995 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:37:13.995 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:37:13.995 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:37:13.995 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:37:13.995 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:37:13.995 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:37:13.996 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:37:13.996 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:37:13.996 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:13.996 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:37:13.996 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:13.996 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:37:13.996 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:37:13.996 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:13.996 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:37:13.996 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T15:37:13.996 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:37:13.996 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:37:13.996 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:37:13.996 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:37:13.997 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:37:13.997 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:37:13.997 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:37:13.997 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:13.997 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:37:13.997 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:13.997 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:37:02.678368+0000", 2024-03-04T15:37:13.997 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:37:13.997 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:13.997 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:37:13.997 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:37:13.997 INFO:teuthology.orchestra.run.smithi005.stdout:mgr not available, waiting (2/15)... 2024-03-04T15:37:14.413 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:14 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:14.148+0000 7f5a54a57200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:37:14.413 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:14 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:14.411+0000 7f5a54a57200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:37:14.680 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:14 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:14.652+0000 7f5a54a57200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:37:15.930 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:15 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:15.473+0000 7f5a54a57200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:37:16.386 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:16.058+0000 7f5a54a57200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:37:16.387 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:16.158+0000 7f5a54a57200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:37:16.387 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:16.384+0000 7f5a54a57200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:37:16.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-mon[25858]: Activating manager daemon y 2024-03-04T15:37:16.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-mon[25858]: mgrmap e2: y(active, starting, since 0.00304434s) 2024-03-04T15:37:16.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-mon[25858]: from='mgr.14100 172.21.15.5:0/2409457289' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:37:16.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-mon[25858]: from='mgr.14100 172.21.15.5:0/2409457289' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:37:16.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-mon[25858]: from='mgr.14100 172.21.15.5:0/2409457289' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:37:16.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-mon[25858]: from='mgr.14100 172.21.15.5:0/2409457289' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:37:16.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-mon[25858]: from='mgr.14100 172.21.15.5:0/2409457289' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:37:16.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-mon[25858]: Manager daemon y is now available 2024-03-04T15:37:16.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-mon[25858]: from='mgr.14100 172.21.15.5:0/2409457289' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:37:16.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-mon[25858]: from='mgr.14100 172.21.15.5:0/2409457289' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:37:16.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-mon[25858]: from='mgr.14100 172.21.15.5:0/2409457289' entity='mgr.y' 2024-03-04T15:37:16.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-mon[25858]: from='mgr.14100 172.21.15.5:0/2409457289' entity='mgr.y' 2024-03-04T15:37:16.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:16 smithi005 ceph-mon[25858]: from='mgr.14100 172.21.15.5:0/2409457289' entity='mgr.y' 2024-03-04T15:37:17.172 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-03-04T15:37:17.172 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:37:17.172 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "fsid": "97c626fc-da3b-11ee-95c6-87774f69a715", 2024-03-04T15:37:17.172 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:37:17.172 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:37:17.173 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:17.174 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:37:17.174 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:17.174 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:37:17.174 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:37:17.174 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:37:17.174 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:37:17.174 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:37:17.174 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:37:17.174 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:17.174 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:37:17.174 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:37:17.174 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:37:17.177 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:37:17.177 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:37:17.177 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:37:17.177 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:37:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:37:17.179 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:37:17.179 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:17.179 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:37:17.179 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:17.179 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:37:02.678368+0000", 2024-03-04T15:37:17.179 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:37:17.179 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:17.179 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:37:17.179 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:37:17.179 INFO:teuthology.orchestra.run.smithi005.stdout:mgr not available, waiting (3/15)... 2024-03-04T15:37:17.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:17 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2237010183' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:37:17.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:17 smithi005 ceph-mon[25858]: mgrmap e3: y(active, since 1.00747s) 2024-03-04T15:37:19.610 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:19 smithi005 ceph-mon[25858]: mgrmap e4: y(active, since 2s) 2024-03-04T15:37:20.497 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:20 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/416861157' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:37:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-03-04T15:37:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:37:20.502 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "fsid": "97c626fc-da3b-11ee-95c6-87774f69a715", 2024-03-04T15:37:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:37:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:37:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:37:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:37:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:37:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:37:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:37:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:37:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:37:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T15:37:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:37:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-04T15:37:20.503 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:37:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T15:37:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:37:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:37:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:37:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:37:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:37:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:37:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:37:20.504 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:37:20.507 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:37:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:37:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:37:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:37:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:37:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:37:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:37:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:37:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:37:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:37:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:37:20.508 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:37:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:37:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T15:37:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:37:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:37:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:37:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:37:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:37:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:37:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:37:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:37:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:37:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:37:02.678368+0000", 2024-03-04T15:37:20.509 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:37:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:37:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:37:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:37:20.510 INFO:teuthology.orchestra.run.smithi005.stdout:mgr is available 2024-03-04T15:37:21.699 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-03-04T15:37:21.699 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout [global] 2024-03-04T15:37:21.699 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout fsid = 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:37:21.699 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-04T15:37:21.699 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-04T15:37:21.700 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-04T15:37:21.700 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-04T15:37:21.700 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-03-04T15:37:21.700 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-04T15:37:21.700 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-04T15:37:21.700 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 2024-03-04T15:37:21.700 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout [osd] 2024-03-04T15:37:21.700 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-04T15:37:21.700 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-04T15:37:21.700 INFO:teuthology.orchestra.run.smithi005.stdout:Enabling cephadm module... 2024-03-04T15:37:21.821 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:21 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2661302952' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-04T15:37:21.821 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:21 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2661302952' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-04T15:37:22.752 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:22 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/3481784520' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-04T15:37:22.753 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:22 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: ignoring --setuser ceph since I am not root 2024-03-04T15:37:22.753 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:22 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: ignoring --setgroup ceph since I am not root 2024-03-04T15:37:23.293 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:23 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:23.031+0000 7f5a26fd8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:37:23.293 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:23 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:23.134+0000 7f5a26fd8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:37:23.680 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:23 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/3481784520' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-04T15:37:23.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:23 smithi005 ceph-mon[25858]: mgrmap e5: y(active, since 6s) 2024-03-04T15:37:24.048 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:23 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:23.806+0000 7f5a26fd8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:37:24.122 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:23 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:23.901+0000 7f5a26fd8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:37:24.123 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:24 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:24.045+0000 7f5a26fd8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:37:24.326 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:24 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:24.144+0000 7f5a26fd8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:37:24.326 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:24 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:24.228+0000 7f5a26fd8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:37:24.329 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:37:24.329 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-04T15:37:24.329 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T15:37:24.329 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-04T15:37:24.329 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-04T15:37:24.329 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:37:24.329 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for the mgr to restart... 2024-03-04T15:37:24.329 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for mgr epoch 5... 2024-03-04T15:37:24.586 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:24 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/1921194732' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-04T15:37:24.586 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:24 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:24.504+0000 7f5a26fd8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:37:24.931 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:24 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:24.619+0000 7f5a26fd8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:37:24.931 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:24 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:24.708+0000 7f5a26fd8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:37:24.931 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:24 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:24.808+0000 7f5a26fd8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:37:25.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:24 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:24.988+0000 7f5a26fd8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:37:26.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:25 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:25.973+0000 7f5a26fd8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:37:26.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:26 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:26.106+0000 7f5a26fd8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:37:26.739 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:26 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:26.445+0000 7f5a26fd8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:37:26.740 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:26 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:26.537+0000 7f5a26fd8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:37:26.740 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:26 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:26.737+0000 7f5a26fd8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:37:27.180 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:26 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:26.926+0000 7f5a26fd8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:37:27.481 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:27 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:27.217+0000 7f5a26fd8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:37:27.481 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:27 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:27.478+0000 7f5a26fd8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:37:27.930 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:27 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:27.720+0000 7f5a26fd8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:37:28.930 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:28 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:28.544+0000 7f5a26fd8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:37:29.430 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:29.127+0000 7f5a26fd8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:37:29.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:29.227+0000 7f5a26fd8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:37:29.931 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:29.447+0000 7f5a26fd8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:37:29.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: Active manager daemon y restarted 2024-03-04T15:37:29.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: Activating manager daemon y 2024-03-04T15:37:29.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: osdmap e2: 0 total, 0 up, 0 in 2024-03-04T15:37:29.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: mgrmap e6: y(active, starting, since 0.00315829s) 2024-03-04T15:37:29.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:37:29.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:37:29.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:37:29.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:37:29.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:37:29.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: Manager daemon y is now available 2024-03-04T15:37:29.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: Found migration_current of "None". Setting to last migration. 2024-03-04T15:37:29.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:29.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:29.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:29.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:37:29.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:29 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:37:30.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:30 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:37:30.732 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:30 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:37:30.732 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:30 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:37:30.732 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:30 smithi005 ceph-mon[25858]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-04T15:37:30.732 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:30 smithi005 ceph-mon[25858]: mgrmap e7: y(active, since 1.00525s) 2024-03-04T15:37:30.732 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:30 smithi005 ceph-mon[25858]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-04T15:37:31.001 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:37:31.002 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-04T15:37:31.002 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-04T15:37:31.002 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:37:31.002 INFO:teuthology.orchestra.run.smithi005.stdout:mgr epoch 5 is available 2024-03-04T15:37:31.002 INFO:teuthology.orchestra.run.smithi005.stdout:Setting orchestrator backend to cephadm... 2024-03-04T15:37:32.330 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:32 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:32.331 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:32 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:32.331 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:32 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:32.331 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:32 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:37:32.331 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:32 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:37:33.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:33 smithi005 ceph-mon[25858]: [04/Mar/2024:15:37:31] ENGINE Bus STARTING 2024-03-04T15:37:33.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:33 smithi005 ceph-mon[25858]: [04/Mar/2024:15:37:31] ENGINE Serving on https://172.21.15.5:7150 2024-03-04T15:37:33.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:33 smithi005 ceph-mon[25858]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:37:33.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:33 smithi005 ceph-mon[25858]: [04/Mar/2024:15:37:31] ENGINE Serving on http://172.21.15.5:8765 2024-03-04T15:37:33.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:33 smithi005 ceph-mon[25858]: [04/Mar/2024:15:37:31] ENGINE Bus STARTED 2024-03-04T15:37:33.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:33 smithi005 ceph-mon[25858]: mgrmap e8: y(active, since 2s) 2024-03-04T15:37:33.430 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-04T15:37:33.431 INFO:teuthology.orchestra.run.smithi005.stdout:Generating ssh key... 2024-03-04T15:37:34.302 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-mon[25858]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:37:34.617 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: Generating public/private rsa key pair. 2024-03-04T15:37:34.617 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: Your identification has been saved in /tmp/tmpa1nwva4v/key. 2024-03-04T15:37:34.617 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: Your public key has been saved in /tmp/tmpa1nwva4v/key.pub. 2024-03-04T15:37:34.617 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: The key fingerprint is: 2024-03-04T15:37:34.617 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: SHA256:yO1hf74683Eopeiud3pI+q1Ix6CXNfnVkd1rRTtYtH8 ceph-97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:37:34.618 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: The key's randomart image is: 2024-03-04T15:37:34.618 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: +---[RSA 3072]----+ 2024-03-04T15:37:34.618 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: | .o.| 2024-03-04T15:37:34.618 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: | o++| 2024-03-04T15:37:34.618 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: | .o++| 2024-03-04T15:37:34.618 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: | . o . . .=| 2024-03-04T15:37:34.618 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: | + S o .oE| 2024-03-04T15:37:34.618 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: | . B.* + .. .| 2024-03-04T15:37:34.618 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: | . +o=.= + . | 2024-03-04T15:37:34.618 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: | o.+o.=+ o | 2024-03-04T15:37:34.618 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: | o=*=o=+. | 2024-03-04T15:37:34.618 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:34 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: +----[SHA256]-----+ 2024-03-04T15:37:35.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:35 smithi005 ceph-mon[25858]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:37:35.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:35 smithi005 ceph-mon[25858]: Generating ssh key... 2024-03-04T15:37:35.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:35 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:35.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:35 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:36.042 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp5JhdtcD1F6cWCe5Q6kC+mwbQRvEtDqx+rzGuUp7JQWUHgOt8d7RkP1YxXOCb+yUFe64CGYZtGZnDxHH8alx6x1IBIu77eEtb8XxvvcFIIcoCJ9eXdTP5nfgvtiRakse2xFG7NYDzfvKdK5QE0LhzNF/L1T8Ouf5B/7r5+Yg2UDvPXBhjYaWgyLN5UQzsjrw3Q33qD9avGcD9rgKhtgeAhSiBEOi0740XocVomgEAtrqDGuICjzQZtQ5usq6fqhfovysjTfozlqfaZEydQHlNvsmoRUst84Us9hJaNFrmwQxlrlmpp+TD+EpSgTSf9xpTAbIidXNPVurLqjXpMuTZn3YtvfQSLnQ1JYDK4lJGq9h2htgAD1tvwBtk7VJv9tbF71wOSvfH79P/hY1uurCHypFIPN34OJ5s3e2spDXf5mqjpsmo4Xm3LIDFkYNIL3FWSmS8fgcAkWUw/Mit4CZPcLekEOx1bzr3+9dQfQbTFjsLb5wa4owDB6NtMrPksZU= ceph-97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:37:36.043 INFO:teuthology.orchestra.run.smithi005.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-04T15:37:36.043 INFO:teuthology.orchestra.run.smithi005.stdout:Adding key to root@localhost authorized_keys... 2024-03-04T15:37:36.043 INFO:teuthology.orchestra.run.smithi005.stdout:Adding host smithi005... 2024-03-04T15:37:36.735 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:36 smithi005 ceph-mon[25858]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:37:37.680 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:37 smithi005 ceph-mon[25858]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi005", "addr": "172.21.15.5", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:37:37.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:37 smithi005 ceph-mon[25858]: Deploying cephadm binary to smithi005 2024-03-04T15:37:41.642 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout Added host 'smithi005' with addr '172.21.15.5' 2024-03-04T15:37:41.643 INFO:teuthology.orchestra.run.smithi005.stdout:Deploying unmanaged mon service... 2024-03-04T15:37:42.347 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:42 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:42.347 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:42 smithi005 ceph-mon[25858]: Added host smithi005 2024-03-04T15:37:42.347 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:42 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:37:43.066 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-04T15:37:43.067 INFO:teuthology.orchestra.run.smithi005.stdout:Deploying unmanaged mgr service... 2024-03-04T15:37:43.599 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:43 smithi005 ceph-mon[25858]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:37:43.600 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:43 smithi005 ceph-mon[25858]: Saving service mon spec with placement count:5 2024-03-04T15:37:43.600 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:43 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:45.293 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:45 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:45.294 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:45 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:45.294 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:45 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:45.295 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-04T15:37:46.362 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:46 smithi005 ceph-mon[25858]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:37:46.362 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:46 smithi005 ceph-mon[25858]: Saving service mgr spec with placement count:2 2024-03-04T15:37:46.362 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:46 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/1949338725' entity='client.admin' 2024-03-04T15:37:47.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:47 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:47.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:47 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:47.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:47 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi005", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:37:47.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:47 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' 2024-03-04T15:37:47.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:47 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi005", "caps": []}]: dispatch 2024-03-04T15:37:47.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:47 smithi005 ceph-mon[25858]: from='mgr.14120 172.21.15.5:0/215809468' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi005", "caps": []}]': finished 2024-03-04T15:37:47.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:47 smithi005 ceph-mon[25858]: Deploying daemon agent.smithi005 on smithi005 2024-03-04T15:37:48.244 INFO:teuthology.orchestra.run.smithi005.stdout:Enabling the dashboard module... 2024-03-04T15:37:48.930 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:48 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/811738192' entity='client.admin' 2024-03-04T15:37:50.048 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:49 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/743434318' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-04T15:37:50.048 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:49 smithi005 ceph-mon[25858]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:37:50.049 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:49 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: ignoring --setuser ceph since I am not root 2024-03-04T15:37:50.049 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:49 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: ignoring --setgroup ceph since I am not root 2024-03-04T15:37:50.559 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:50 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:50.315+0000 7f8acba87200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:37:50.559 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:50 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:50.415+0000 7f8acba87200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:37:51.080 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:50 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/743434318' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-04T15:37:51.080 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:50 smithi005 ceph-mon[25858]: mgrmap e9: y(active, since 20s) 2024-03-04T15:37:51.332 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:51 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:51.089+0000 7f8acba87200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:37:51.332 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:51 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:51.186+0000 7f8acba87200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:37:51.332 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:51 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:51.330+0000 7f8acba87200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:37:51.567 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:37:51.567 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-04T15:37:51.567 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T15:37:51.567 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-04T15:37:51.567 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-04T15:37:51.567 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:37:51.567 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for the mgr to restart... 2024-03-04T15:37:51.567 INFO:teuthology.orchestra.run.smithi005.stdout:Waiting for mgr epoch 9... 2024-03-04T15:37:51.680 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:51 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:51.429+0000 7f8acba87200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:37:51.681 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:51 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:51.514+0000 7f8acba87200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:37:51.958 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:51 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/975203770' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-04T15:37:51.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:51 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:51.793+0000 7f8acba87200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:37:51.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:51 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:51.908+0000 7f8acba87200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:37:52.277 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:52 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:52.000+0000 7f8acba87200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:37:52.277 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:52 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:52.097+0000 7f8acba87200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:37:52.277 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:52 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:52.275+0000 7f8acba87200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:37:53.680 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:53 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:53.263+0000 7f8acba87200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:37:53.681 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:53 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:53.397+0000 7f8acba87200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:37:54.032 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:53 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:53.733+0000 7f8acba87200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:37:54.032 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:53 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:53.827+0000 7f8acba87200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:37:54.032 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:54 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:54.030+0000 7f8acba87200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:37:54.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:54 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:54.221+0000 7f8acba87200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:37:54.779 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:54 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:54.516+0000 7f8acba87200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:37:54.784 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:54 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:54.777+0000 7f8acba87200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:37:55.180 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:55 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:55.022+0000 7f8acba87200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:37:56.181 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:55 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:55.853+0000 7f8acba87200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:37:56.764 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:56.435+0000 7f8acba87200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:37:56.764 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:56.538+0000 7f8acba87200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:37:56.765 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:37:56.762+0000 7f8acba87200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:37:57.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-mon[25858]: Active manager daemon y restarted 2024-03-04T15:37:57.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-mon[25858]: Activating manager daemon y 2024-03-04T15:37:57.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-mon[25858]: osdmap e3: 0 total, 0 up, 0 in 2024-03-04T15:37:57.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-mon[25858]: mgrmap e10: y(active, starting, since 0.00319653s) 2024-03-04T15:37:57.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:37:57.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:37:57.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:37:57.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:37:57.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:37:57.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-mon[25858]: Manager daemon y is now available 2024-03-04T15:37:57.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:37:57.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:37:57.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:56 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:37:58.112 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:37:58.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:37:58.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:57 smithi005 ceph-mon[25858]: [04/Mar/2024:15:37:57] ENGINE Bus STARTING 2024-03-04T15:37:58.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:57 smithi005 ceph-mon[25858]: [04/Mar/2024:15:37:57] ENGINE Serving on http://172.21.15.5:8765 2024-03-04T15:37:58.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:57 smithi005 ceph-mon[25858]: [04/Mar/2024:15:37:57] ENGINE Serving on https://172.21.15.5:7150 2024-03-04T15:37:58.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:57 smithi005 ceph-mon[25858]: [04/Mar/2024:15:37:57] ENGINE Bus STARTED 2024-03-04T15:37:58.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:57 smithi005 ceph-mon[25858]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-04T15:37:58.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:57 smithi005 ceph-mon[25858]: mgrmap e11: y(active, since 1.00609s) 2024-03-04T15:37:58.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:57 smithi005 ceph-mon[25858]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-04T15:37:58.498 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:37:58.499 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-04T15:37:58.499 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-04T15:37:58.499 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:37:58.499 INFO:teuthology.orchestra.run.smithi005.stdout:mgr epoch 9 is available 2024-03-04T15:37:58.499 INFO:teuthology.orchestra.run.smithi005.stdout:Generating a dashboard self-signed certificate... 2024-03-04T15:37:59.831 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-04T15:37:59.831 INFO:teuthology.orchestra.run.smithi005.stdout:Creating initial admin user... 2024-03-04T15:37:59.930 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:59 smithi005 ceph-mon[25858]: mgrmap e12: y(active, since 2s) 2024-03-04T15:37:59.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:59 smithi005 ceph-mon[25858]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:37:59.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:59 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:37:59.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:37:59 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:01.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:00 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:01.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:00 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi005", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:38:01.118 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:00 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:01.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:00 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi005", "caps": []}]: dispatch 2024-03-04T15:38:01.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:00 smithi005 ceph-mon[25858]: Deploying daemon agent.smithi005 on smithi005 2024-03-04T15:38:01.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:00 smithi005 ceph-mon[25858]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:38:01.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:00 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:01.413 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$dmG7PmPC6qEmye5z.mlIPe.6geUNgjw0m78BAwF4jgP663AZG78h2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1709566680, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-04T15:38:01.413 INFO:teuthology.orchestra.run.smithi005.stdout:Fetching dashboard port number... 2024-03-04T15:38:02.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:02 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/1412356564' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-04T15:38:02.660 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stdout 8443 2024-03-04T15:38:02.669 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:38:02.669 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout disabled 2024-03-04T15:38:02.676 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:38:02.676 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stdout inactive 2024-03-04T15:38:02.676 INFO:teuthology.orchestra.run.smithi005.stdout:firewalld.service is not enabled 2024-03-04T15:38:02.676 INFO:teuthology.orchestra.run.smithi005.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-04T15:38:02.678 INFO:teuthology.orchestra.run.smithi005.stdout:Ceph Dashboard is now available at: 2024-03-04T15:38:02.678 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:38:02.678 INFO:teuthology.orchestra.run.smithi005.stdout: URL: https://smithi005.front.sepia.ceph.com:8443/ 2024-03-04T15:38:02.679 INFO:teuthology.orchestra.run.smithi005.stdout: User: admin 2024-03-04T15:38:02.679 INFO:teuthology.orchestra.run.smithi005.stdout: Password: bvamaiun4d 2024-03-04T15:38:02.679 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:38:02.679 INFO:teuthology.orchestra.run.smithi005.stdout:Saving cluster configuration to /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config directory 2024-03-04T15:38:02.679 INFO:teuthology.orchestra.run.smithi005.stdout:Enabling autotune for osd_memory_target 2024-03-04T15:38:04.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:04 smithi005 ceph-mon[25858]: mgrmap e13: y(active, since 6s) 2024-03-04T15:38:05.273 INFO:teuthology.orchestra.run.smithi005.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-04T15:38:05.273 INFO:teuthology.orchestra.run.smithi005.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-04T15:38:05.273 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:38:05.273 INFO:teuthology.orchestra.run.smithi005.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:38:05.273 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:38:05.274 INFO:teuthology.orchestra.run.smithi005.stdout:Or, if you are only running a single cluster on this host: 2024-03-04T15:38:05.274 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:38:05.274 INFO:teuthology.orchestra.run.smithi005.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-04T15:38:05.274 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:38:05.274 INFO:teuthology.orchestra.run.smithi005.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-04T15:38:05.274 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:38:05.274 INFO:teuthology.orchestra.run.smithi005.stdout: ceph telemetry on 2024-03-04T15:38:05.274 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:38:05.274 INFO:teuthology.orchestra.run.smithi005.stdout:For more information see: 2024-03-04T15:38:05.274 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:38:05.274 INFO:teuthology.orchestra.run.smithi005.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-04T15:38:05.274 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:38:05.274 INFO:teuthology.orchestra.run.smithi005.stdout:Bootstrap complete. 2024-03-04T15:38:05.317 INFO:tasks.cephadm:Fetching config... 2024-03-04T15:38:05.318 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-04T15:38:05.318 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-04T15:38:05.333 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-04T15:38:05.333 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-04T15:38:05.333 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-04T15:38:05.388 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-04T15:38:05.388 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-04T15:38:05.388 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-04T15:38:05.570 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-04T15:38:05.570 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-04T15:38:05.570 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-04T15:38:05.626 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-04T15:38:05.626 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp5JhdtcD1F6cWCe5Q6kC+mwbQRvEtDqx+rzGuUp7JQWUHgOt8d7RkP1YxXOCb+yUFe64CGYZtGZnDxHH8alx6x1IBIu77eEtb8XxvvcFIIcoCJ9eXdTP5nfgvtiRakse2xFG7NYDzfvKdK5QE0LhzNF/L1T8Ouf5B/7r5+Yg2UDvPXBhjYaWgyLN5UQzsjrw3Q33qD9avGcD9rgKhtgeAhSiBEOi0740XocVomgEAtrqDGuICjzQZtQ5usq6fqhfovysjTfozlqfaZEydQHlNvsmoRUst84Us9hJaNFrmwQxlrlmpp+TD+EpSgTSf9xpTAbIidXNPVurLqjXpMuTZn3YtvfQSLnQ1JYDK4lJGq9h2htgAD1tvwBtk7VJv9tbF71wOSvfH79P/hY1uurCHypFIPN34OJ5s3e2spDXf5mqjpsmo4Xm3LIDFkYNIL3FWSmS8fgcAkWUw/Mit4CZPcLekEOx1bzr3+9dQfQbTFjsLb5wa4owDB6NtMrPksZU= ceph-97c626fc-da3b-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T15:38:05.779 INFO:teuthology.orchestra.run.smithi005.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp5JhdtcD1F6cWCe5Q6kC+mwbQRvEtDqx+rzGuUp7JQWUHgOt8d7RkP1YxXOCb+yUFe64CGYZtGZnDxHH8alx6x1IBIu77eEtb8XxvvcFIIcoCJ9eXdTP5nfgvtiRakse2xFG7NYDzfvKdK5QE0LhzNF/L1T8Ouf5B/7r5+Yg2UDvPXBhjYaWgyLN5UQzsjrw3Q33qD9avGcD9rgKhtgeAhSiBEOi0740XocVomgEAtrqDGuICjzQZtQ5usq6fqhfovysjTfozlqfaZEydQHlNvsmoRUst84Us9hJaNFrmwQxlrlmpp+TD+EpSgTSf9xpTAbIidXNPVurLqjXpMuTZn3YtvfQSLnQ1JYDK4lJGq9h2htgAD1tvwBtk7VJv9tbF71wOSvfH79P/hY1uurCHypFIPN34OJ5s3e2spDXf5mqjpsmo4Xm3LIDFkYNIL3FWSmS8fgcAkWUw/Mit4CZPcLekEOx1bzr3+9dQfQbTFjsLb5wa4owDB6NtMrPksZU= ceph-97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:38:05.793 DEBUG:teuthology.orchestra.run.smithi080:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp5JhdtcD1F6cWCe5Q6kC+mwbQRvEtDqx+rzGuUp7JQWUHgOt8d7RkP1YxXOCb+yUFe64CGYZtGZnDxHH8alx6x1IBIu77eEtb8XxvvcFIIcoCJ9eXdTP5nfgvtiRakse2xFG7NYDzfvKdK5QE0LhzNF/L1T8Ouf5B/7r5+Yg2UDvPXBhjYaWgyLN5UQzsjrw3Q33qD9avGcD9rgKhtgeAhSiBEOi0740XocVomgEAtrqDGuICjzQZtQ5usq6fqhfovysjTfozlqfaZEydQHlNvsmoRUst84Us9hJaNFrmwQxlrlmpp+TD+EpSgTSf9xpTAbIidXNPVurLqjXpMuTZn3YtvfQSLnQ1JYDK4lJGq9h2htgAD1tvwBtk7VJv9tbF71wOSvfH79P/hY1uurCHypFIPN34OJ5s3e2spDXf5mqjpsmo4Xm3LIDFkYNIL3FWSmS8fgcAkWUw/Mit4CZPcLekEOx1bzr3+9dQfQbTFjsLb5wa4owDB6NtMrPksZU= ceph-97c626fc-da3b-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T15:38:05.825 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:05 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/1411524123' entity='client.admin' 2024-03-04T15:38:05.839 INFO:teuthology.orchestra.run.smithi080.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp5JhdtcD1F6cWCe5Q6kC+mwbQRvEtDqx+rzGuUp7JQWUHgOt8d7RkP1YxXOCb+yUFe64CGYZtGZnDxHH8alx6x1IBIu77eEtb8XxvvcFIIcoCJ9eXdTP5nfgvtiRakse2xFG7NYDzfvKdK5QE0LhzNF/L1T8Ouf5B/7r5+Yg2UDvPXBhjYaWgyLN5UQzsjrw3Q33qD9avGcD9rgKhtgeAhSiBEOi0740XocVomgEAtrqDGuICjzQZtQ5usq6fqhfovysjTfozlqfaZEydQHlNvsmoRUst84Us9hJaNFrmwQxlrlmpp+TD+EpSgTSf9xpTAbIidXNPVurLqjXpMuTZn3YtvfQSLnQ1JYDK4lJGq9h2htgAD1tvwBtk7VJv9tbF71wOSvfH79P/hY1uurCHypFIPN34OJ5s3e2spDXf5mqjpsmo4Xm3LIDFkYNIL3FWSmS8fgcAkWUw/Mit4CZPcLekEOx1bzr3+9dQfQbTFjsLb5wa4owDB6NtMrPksZU= ceph-97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:38:05.854 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp5JhdtcD1F6cWCe5Q6kC+mwbQRvEtDqx+rzGuUp7JQWUHgOt8d7RkP1YxXOCb+yUFe64CGYZtGZnDxHH8alx6x1IBIu77eEtb8XxvvcFIIcoCJ9eXdTP5nfgvtiRakse2xFG7NYDzfvKdK5QE0LhzNF/L1T8Ouf5B/7r5+Yg2UDvPXBhjYaWgyLN5UQzsjrw3Q33qD9avGcD9rgKhtgeAhSiBEOi0740XocVomgEAtrqDGuICjzQZtQ5usq6fqhfovysjTfozlqfaZEydQHlNvsmoRUst84Us9hJaNFrmwQxlrlmpp+TD+EpSgTSf9xpTAbIidXNPVurLqjXpMuTZn3YtvfQSLnQ1JYDK4lJGq9h2htgAD1tvwBtk7VJv9tbF71wOSvfH79P/hY1uurCHypFIPN34OJ5s3e2spDXf5mqjpsmo4Xm3LIDFkYNIL3FWSmS8fgcAkWUw/Mit4CZPcLekEOx1bzr3+9dQfQbTFjsLb5wa4owDB6NtMrPksZU= ceph-97c626fc-da3b-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T15:38:05.902 INFO:teuthology.orchestra.run.smithi087.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp5JhdtcD1F6cWCe5Q6kC+mwbQRvEtDqx+rzGuUp7JQWUHgOt8d7RkP1YxXOCb+yUFe64CGYZtGZnDxHH8alx6x1IBIu77eEtb8XxvvcFIIcoCJ9eXdTP5nfgvtiRakse2xFG7NYDzfvKdK5QE0LhzNF/L1T8Ouf5B/7r5+Yg2UDvPXBhjYaWgyLN5UQzsjrw3Q33qD9avGcD9rgKhtgeAhSiBEOi0740XocVomgEAtrqDGuICjzQZtQ5usq6fqhfovysjTfozlqfaZEydQHlNvsmoRUst84Us9hJaNFrmwQxlrlmpp+TD+EpSgTSf9xpTAbIidXNPVurLqjXpMuTZn3YtvfQSLnQ1JYDK4lJGq9h2htgAD1tvwBtk7VJv9tbF71wOSvfH79P/hY1uurCHypFIPN34OJ5s3e2spDXf5mqjpsmo4Xm3LIDFkYNIL3FWSmS8fgcAkWUw/Mit4CZPcLekEOx1bzr3+9dQfQbTFjsLb5wa4owDB6NtMrPksZU= ceph-97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:38:05.917 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-04T15:38:06.311 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:38:07.601 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:07 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:07.601 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:07 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:07.601 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:07 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:07.601 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:07 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:07.601 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:07 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:07.601 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:07 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:07.601 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:07 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:07.602 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:07 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:07.602 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:07 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:07.602 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:07 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:07.602 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:07 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:07.602 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:07 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:07.602 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:07 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:09.082 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-04T15:38:09.082 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-04T15:38:09.599 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:09 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/4172612400' entity='client.admin' 2024-03-04T15:38:09.600 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:09 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:09.600 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:09 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:38:11.288 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:10 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:11.288 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:10 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:12.793 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi080 2024-03-04T15:38:12.793 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-04T15:38:12.793 DEBUG:teuthology.orchestra.run.smithi080:> dd of=/etc/ceph/ceph.conf 2024-03-04T15:38:12.813 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-04T15:38:12.813 DEBUG:teuthology.orchestra.run.smithi080:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:38:12.869 INFO:tasks.cephadm:Adding host smithi080 to orchestrator... 2024-03-04T15:38:12.869 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph orch host add smithi080 2024-03-04T15:38:13.248 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:38:13.336 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:13 smithi005 ceph-mon[25858]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:38:13.336 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:13 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:13.336 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:13 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:13.336 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:13 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:13.336 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:13 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:13.336 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:13 smithi005 ceph-mon[25858]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-04T15:38:13.336 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:13 smithi005 ceph-mon[25858]: Updating smithi005:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:38:13.336 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:13 smithi005 ceph-mon[25858]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:38:13.336 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:13 smithi005 ceph-mon[25858]: Updating smithi005:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:38:13.336 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:13 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:13.337 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:13 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:14.642 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:14.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:16.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:15 smithi005 ceph-mon[25858]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi080", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:38:16.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:15 smithi005 ceph-mon[25858]: Deploying cephadm binary to smithi080 2024-03-04T15:38:17.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:16 smithi005 ceph-mon[25858]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:19.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:18 smithi005 ceph-mon[25858]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:19.392 INFO:teuthology.orchestra.run.smithi005.stdout:Added host 'smithi080' with addr '172.21.15.80' 2024-03-04T15:38:20.143 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph orch host ls --format=json 2024-03-04T15:38:20.394 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:38:20.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:20 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:20.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:20 smithi005 ceph-mon[25858]: Added host smithi080 2024-03-04T15:38:20.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:20 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:20.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:20 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:21.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:21 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:21.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:21 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:21.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:21 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi080", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:38:21.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:21 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:21.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:21 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:21.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:21 smithi005 ceph-mon[25858]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-04T15:38:21.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:21 smithi005 ceph-mon[25858]: Updating smithi080:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:38:21.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:21 smithi005 ceph-mon[25858]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:21.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:21 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:21.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:21 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:21.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:21 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi080", "caps": []}]: dispatch 2024-03-04T15:38:21.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:21 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi080", "caps": []}]': finished 2024-03-04T15:38:22.207 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:38:22.207 INFO:teuthology.orchestra.run.smithi005.stdout:[{"addr": "172.21.15.5", "hostname": "smithi005", "labels": [], "status": ""}, {"addr": "172.21.15.80", "hostname": "smithi080", "labels": [], "status": ""}] 2024-03-04T15:38:22.517 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:22 smithi005 ceph-mon[25858]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:38:22.517 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:22 smithi005 ceph-mon[25858]: Updating smithi080:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:38:22.518 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:22 smithi005 ceph-mon[25858]: Deploying daemon agent.smithi080 on smithi080 2024-03-04T15:38:22.876 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi087 2024-03-04T15:38:22.876 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-04T15:38:22.876 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/etc/ceph/ceph.conf 2024-03-04T15:38:22.894 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-04T15:38:22.895 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:38:22.951 INFO:tasks.cephadm:Adding host smithi087 to orchestrator... 2024-03-04T15:38:22.952 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph orch host add smithi087 2024-03-04T15:38:23.187 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:38:23.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:38:23.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:23.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:23.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:23.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:23.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:23.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:23.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:23.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:23.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:23.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:23.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:23.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:23.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:23.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:23.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:25.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:24 smithi005 ceph-mon[25858]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:26.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:25 smithi005 ceph-mon[25858]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi087", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:38:26.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:25 smithi005 ceph-mon[25858]: Deploying cephadm binary to smithi087 2024-03-04T15:38:27.546 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:27 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:27.546 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:27 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:27.546 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:27 smithi005 ceph-mon[25858]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:27.546 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:27 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:29.337 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:29 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:29.337 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:29 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:29.337 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:29 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:29.337 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:29 smithi005 ceph-mon[25858]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:29.338 INFO:teuthology.orchestra.run.smithi005.stdout:Added host 'smithi087' with addr '172.21.15.87' 2024-03-04T15:38:30.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:30 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:30.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:30 smithi005 ceph-mon[25858]: Added host smithi087 2024-03-04T15:38:30.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:30 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:30.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:30 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:30.598 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph orch host ls --format=json 2024-03-04T15:38:30.842 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:38:31.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:31.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:31.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi087", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:38:31.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:31.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:31.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:31 smithi005 ceph-mon[25858]: Updating smithi087:/etc/ceph/ceph.conf 2024-03-04T15:38:31.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:31 smithi005 ceph-mon[25858]: Updating smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:38:31.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:31 smithi005 ceph-mon[25858]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:31.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:31.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:31.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi087", "caps": []}]: dispatch 2024-03-04T15:38:31.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi087", "caps": []}]': finished 2024-03-04T15:38:32.673 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:38:32.673 INFO:teuthology.orchestra.run.smithi005.stdout:[{"addr": "172.21.15.5", "hostname": "smithi005", "labels": [], "status": ""}, {"addr": "172.21.15.80", "hostname": "smithi080", "labels": [], "status": ""}, {"addr": "172.21.15.87", "hostname": "smithi087", "labels": [], "status": ""}] 2024-03-04T15:38:32.673 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:32 smithi005 ceph-mon[25858]: Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:38:32.673 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:32 smithi005 ceph-mon[25858]: Updating smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:38:33.366 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-04T15:38:33.366 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph osd crush tunables default 2024-03-04T15:38:33.601 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:38:33.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:33 smithi005 ceph-mon[25858]: Deploying daemon agent.smithi087 on smithi087 2024-03-04T15:38:33.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:33 smithi005 ceph-mon[25858]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:38:33.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:33 smithi005 ceph-mon[25858]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:33.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:33.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:33.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:33.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:33.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:33.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:33.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:33.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:33.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:33.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:33.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:33.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:34.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:34.656 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:35.626 INFO:teuthology.orchestra.run.smithi005.stderr:adjusted tunables profile to default 2024-03-04T15:38:35.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:35 smithi005 ceph-mon[25858]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:35.910 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:35 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/4051206117' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-04T15:38:36.164 INFO:tasks.cephadm:Adding mon.a on smithi005 2024-03-04T15:38:36.164 INFO:tasks.cephadm:Adding mon.b on smithi080 2024-03-04T15:38:36.164 INFO:tasks.cephadm:Adding mon.c on smithi087 2024-03-04T15:38:36.164 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph orch apply mon '3;smithi005:172.21.15.5=a;smithi080:172.21.15.80=b;smithi087:172.21.15.87=c' 2024-03-04T15:38:36.398 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:38:36.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:36 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/4051206117' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-04T15:38:36.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:36 smithi005 ceph-mon[25858]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:38:36.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:36 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:36.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:36 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:37.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:37 smithi005 ceph-mon[25858]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:37.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:37 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:38.136 INFO:teuthology.orchestra.run.smithi087.stdout:Scheduled mon update... 2024-03-04T15:38:38.751 DEBUG:teuthology.orchestra.run.smithi080:mon.b> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mon.b.service 2024-03-04T15:38:38.754 DEBUG:teuthology.orchestra.run.smithi087:mon.c> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mon.c.service 2024-03-04T15:38:38.756 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T15:38:38.757 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:38:39.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:39 smithi005 ceph-mon[25858]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi005:172.21.15.5=a;smithi080:172.21.15.80=b;smithi087:172.21.15.87=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:38:39.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:39 smithi005 ceph-mon[25858]: Saving service mon spec with placement smithi005:172.21.15.5=a;smithi080:172.21.15.80=b;smithi087:172.21.15.87=c;count:3 2024-03-04T15:38:39.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:39 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:39.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:39 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:39.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:39 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:39.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:39 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:39.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:39 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:39.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:39 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:38:39.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:39 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:39.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:39 smithi005 ceph-mon[25858]: Deploying daemon mon.c on smithi087 2024-03-04T15:38:39.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:39 smithi005 ceph-mon[25858]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:39.856 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:38:40.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:40 smithi005 ceph-mon[25858]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:42.833 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-04T15:38:42.833 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":1,"fsid":"97c626fc-da3b-11ee-95c6-87774f69a715","modified":"2024-03-04T15:36:59.813782Z","created":"2024-03-04T15:36:59.813782Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:38:42.835 INFO:teuthology.orchestra.run.smithi087.stderr:dumped monmap epoch 1 2024-03-04T15:38:43.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:42 smithi005 ceph-mon[25858]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:43.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:42 smithi005 ceph-mon[25858]: from='client.? 172.21.15.87:0/2972819078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:38:43.718 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 podman[27627]: 2024-03-04 15:38:43.559156973 +0000 UTC m=+0.018423262 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:38:44.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 podman[27627]: 2024-03-04 15:38:43.804337303 +0000 UTC m=+0.263603564 container init 22502b7796bd1d7de21140b603c97f5dba36dbf115b0db40a94a13df392113a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-03-04T15:38:44.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 podman[27627]: 2024-03-04 15:38:43.809637885 +0000 UTC m=+0.268904144 container start 22502b7796bd1d7de21140b603c97f5dba36dbf115b0db40a94a13df392113a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-c, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-04T15:38:44.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T15:38:44.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev), process ceph-mon, pid 7 2024-03-04T15:38:44.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: pidfile_write: ignore empty --pid-file 2024-03-04T15:38:44.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: load: jerasure load: lrc 2024-03-04T15:38:44.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: RocksDB version: 7.9.2 2024-03-04T15:38:44.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Git sha 0 2024-03-04T15:38:44.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Compile date 2024-03-04 11:37:16 2024-03-04T15:38:44.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: DB SUMMARY 2024-03-04T15:38:44.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: DB Session ID: 8MWVODBJ9FQYEBA1MOW2 2024-03-04T15:38:44.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: CURRENT file: CURRENT 2024-03-04T15:38:44.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T15:38:44.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-04T15:38:44.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-04T15:38:44.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-04T15:38:44.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.error_if_exists: 0 2024-03-04T15:38:44.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.create_if_missing: 0 2024-03-04T15:38:44.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.paranoid_checks: 1 2024-03-04T15:38:44.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T15:38:44.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T15:38:44.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T15:38:44.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.env: 0x55e3f3884b20 2024-03-04T15:38:44.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T15:38:44.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.info_log: 0x55e3f51a98e0 2024-03-04T15:38:44.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T15:38:44.282 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.statistics: (nil) 2024-03-04T15:38:44.283 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.use_fsync: 0 2024-03-04T15:38:44.283 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_log_file_size: 0 2024-03-04T15:38:44.283 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T15:38:44.283 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T15:38:44.283 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T15:38:44.283 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T15:38:44.283 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.allow_fallocate: 1 2024-03-04T15:38:44.283 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T15:38:44.283 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T15:38:44.283 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.use_direct_reads: 0 2024-03-04T15:38:44.283 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T15:38:44.283 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T15:38:44.283 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.db_log_dir: 2024-03-04T15:38:44.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.wal_dir: 2024-03-04T15:38:44.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T15:38:44.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T15:38:44.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T15:38:44.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T15:38:44.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T15:38:44.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T15:38:44.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T15:38:44.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T15:38:44.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.write_buffer_manager: 0x55e3f517d400 2024-03-04T15:38:44.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T15:38:44.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T15:38:44.285 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T15:38:44.285 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T15:38:44.285 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T15:38:44.285 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T15:38:44.285 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T15:38:44.285 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T15:38:44.285 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.unordered_write: 0 2024-03-04T15:38:44.285 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T15:38:44.285 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T15:38:44.285 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T15:38:44.285 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T15:38:44.285 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.row_cache: None 2024-03-04T15:38:44.285 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.wal_filter: None 2024-03-04T15:38:44.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T15:38:44.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T15:38:44.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.two_write_queues: 0 2024-03-04T15:38:44.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T15:38:44.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.wal_compression: 0 2024-03-04T15:38:44.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.atomic_flush: 0 2024-03-04T15:38:44.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T15:38:44.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T15:38:44.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T15:38:44.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.log_readahead_size: 0 2024-03-04T15:38:44.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T15:38:44.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T15:38:44.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T15:38:44.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T15:38:44.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T15:38:44.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T15:38:44.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T15:38:44.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_background_jobs: 2 2024-03-04T15:38:44.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_background_compactions: -1 2024-03-04T15:38:44.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_subcompactions: 1 2024-03-04T15:38:44.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T15:38:44.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T15:38:44.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T15:38:44.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T15:38:44.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T15:38:44.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T15:38:44.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T15:38:44.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T15:38:44.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_open_files: -1 2024-03-04T15:38:44.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T15:38:44.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T15:38:44.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T15:38:44.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T15:38:44.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_background_flushes: -1 2024-03-04T15:38:44.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Compression algorithms supported: 2024-03-04T15:38:44.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T15:38:44.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: kZSTD supported: 0 2024-03-04T15:38:44.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: kXpressCompression supported: 0 2024-03-04T15:38:44.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T15:38:44.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: kZlibCompression supported: 1 2024-03-04T15:38:44.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: kSnappyCompression supported: 1 2024-03-04T15:38:44.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: kLZ4Compression supported: 1 2024-03-04T15:38:44.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: kBZip2Compression supported: 0 2024-03-04T15:38:44.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T15:38:44.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T15:38:44.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-04T15:38:44.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T15:38:44.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T15:38:44.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.merge_operator: 2024-03-04T15:38:44.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compaction_filter: None 2024-03-04T15:38:44.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T15:38:44.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T15:38:44.290 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T15:38:44.290 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T15:38:44.290 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e3f51a99e0) 2024-03-04T15:38:44.290 INFO:journalctl@ceph.mon.c.smithi087.stdout: cache_index_and_filter_blocks: 1 2024-03-04T15:38:44.290 INFO:journalctl@ceph.mon.c.smithi087.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T15:38:44.290 INFO:journalctl@ceph.mon.c.smithi087.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T15:38:44.290 INFO:journalctl@ceph.mon.c.smithi087.stdout: pin_top_level_index_and_filter: 1 2024-03-04T15:38:44.290 INFO:journalctl@ceph.mon.c.smithi087.stdout: index_type: 0 2024-03-04T15:38:44.290 INFO:journalctl@ceph.mon.c.smithi087.stdout: data_block_index_type: 0 2024-03-04T15:38:44.290 INFO:journalctl@ceph.mon.c.smithi087.stdout: index_shortening: 1 2024-03-04T15:38:44.290 INFO:journalctl@ceph.mon.c.smithi087.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T15:38:44.290 INFO:journalctl@ceph.mon.c.smithi087.stdout: checksum: 4 2024-03-04T15:38:44.291 INFO:journalctl@ceph.mon.c.smithi087.stdout: no_block_cache: 0 2024-03-04T15:38:44.291 INFO:journalctl@ceph.mon.c.smithi087.stdout: block_cache: 0x55e3f51af090 2024-03-04T15:38:44.291 INFO:journalctl@ceph.mon.c.smithi087.stdout: block_cache_name: BinnedLRUCache 2024-03-04T15:38:44.291 INFO:journalctl@ceph.mon.c.smithi087.stdout: block_cache_options: 2024-03-04T15:38:44.291 INFO:journalctl@ceph.mon.c.smithi087.stdout: capacity : 536870912 2024-03-04T15:38:44.291 INFO:journalctl@ceph.mon.c.smithi087.stdout: num_shard_bits : 4 2024-03-04T15:38:44.291 INFO:journalctl@ceph.mon.c.smithi087.stdout: strict_capacity_limit : 0 2024-03-04T15:38:44.291 INFO:journalctl@ceph.mon.c.smithi087.stdout: high_pri_pool_ratio: 0.000 2024-03-04T15:38:44.291 INFO:journalctl@ceph.mon.c.smithi087.stdout: block_cache_compressed: (nil) 2024-03-04T15:38:44.291 INFO:journalctl@ceph.mon.c.smithi087.stdout: persistent_cache: (nil) 2024-03-04T15:38:44.291 INFO:journalctl@ceph.mon.c.smithi087.stdout: block_size: 4096 2024-03-04T15:38:44.291 INFO:journalctl@ceph.mon.c.smithi087.stdout: block_size_deviation: 10 2024-03-04T15:38:44.291 INFO:journalctl@ceph.mon.c.smithi087.stdout: block_restart_interval: 16 2024-03-04T15:38:44.291 INFO:journalctl@ceph.mon.c.smithi087.stdout: index_block_restart_interval: 1 2024-03-04T15:38:44.291 INFO:journalctl@ceph.mon.c.smithi087.stdout: metadata_block_size: 4096 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout: partition_filters: 0 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout: use_delta_encoding: 1 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout: filter_policy: bloomfilter 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout: whole_key_filtering: 1 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout: verify_compression: 0 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout: read_amp_bytes_per_bit: 0 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout: format_version: 5 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout: enable_index_compression: 1 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout: block_align: 0 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout: max_auto_readahead_size: 262144 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout: prepopulate_block_cache: 0 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout: initial_auto_readahead_size: 8192 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T15:38:44.292 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compression: NoCompression 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.num_levels: 7 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:38:44.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T15:38:44.294 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T15:38:44.294 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T15:38:44.294 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T15:38:44.294 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:38:44.294 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:38:44.294 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T15:38:44.294 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T15:38:44.294 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:38:44.294 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T15:38:44.294 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T15:38:44.294 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T15:38:44.294 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T15:38:44.294 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T15:38:44.294 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T15:38:44.295 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T15:38:44.296 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T15:38:44.296 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T15:38:44.296 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T15:38:44.296 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T15:38:44.296 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T15:38:44.296 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T15:38:44.296 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T15:38:44.296 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T15:38:44.296 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T15:38:44.296 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.table_properties_collectors: 2024-03-04T15:38:44.296 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.inplace_update_support: 0 2024-03-04T15:38:44.296 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T15:38:44.296 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T15:38:44.296 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.bloom_locality: 0 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.max_successive_merges: 0 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.ttl: 2592000 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.enable_blob_files: false 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.min_blob_size: 0 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T15:38:44.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T15:38:44.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T15:38:44.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T15:38:44.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T15:38:44.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-04T15:38:44.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-04T15:38:44.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0e2bf7a7-068d-49c7-8477-8941f3d13aaf 2024-03-04T15:38:44.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709566723838823, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-04T15:38:44.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-04T15:38:44.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709566723839239, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709566723, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0e2bf7a7-068d-49c7-8477-8941f3d13aaf", "db_session_id": "8MWVODBJ9FQYEBA1MOW2", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-04T15:38:44.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709566723839318, "job": 1, "event": "recovery_finished"} 2024-03-04T15:38:44.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-04T15:38:44.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T15:38:44.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e3f52a2000 2024-03-04T15:38:44.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: DB pointer 0x55e3f528c000 2024-03-04T15:38:44.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: ** DB Stats ** 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: ** Compaction Stats [default] ** 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:38:44.299 INFO:journalctl@ceph.mon.c.smithi087.stdout: 2024-03-04T15:38:44.300 INFO:journalctl@ceph.mon.c.smithi087.stdout: ** Compaction Stats [default] ** 2024-03-04T15:38:44.300 INFO:journalctl@ceph.mon.c.smithi087.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:38:44.300 INFO:journalctl@ceph.mon.c.smithi087.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:38:44.300 INFO:journalctl@ceph.mon.c.smithi087.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:38:44.300 INFO:journalctl@ceph.mon.c.smithi087.stdout: 2024-03-04T15:38:44.300 INFO:journalctl@ceph.mon.c.smithi087.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T15:38:44.300 INFO:journalctl@ceph.mon.c.smithi087.stdout: 2024-03-04T15:38:44.300 INFO:journalctl@ceph.mon.c.smithi087.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:38:44.300 INFO:journalctl@ceph.mon.c.smithi087.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T15:38:44.300 INFO:journalctl@ceph.mon.c.smithi087.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T15:38:44.300 INFO:journalctl@ceph.mon.c.smithi087.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T15:38:44.300 INFO:journalctl@ceph.mon.c.smithi087.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T15:38:44.300 INFO:journalctl@ceph.mon.c.smithi087.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T15:38:44.300 INFO:journalctl@ceph.mon.c.smithi087.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:38:44.300 INFO:journalctl@ceph.mon.c.smithi087.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:38:44.301 INFO:journalctl@ceph.mon.c.smithi087.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T15:38:44.301 INFO:journalctl@ceph.mon.c.smithi087.stdout: Block cache BinnedLRUCache@0x55e3f51af090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-04T15:38:44.301 INFO:journalctl@ceph.mon.c.smithi087.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-04T15:38:44.301 INFO:journalctl@ceph.mon.c.smithi087.stdout: 2024-03-04T15:38:44.301 INFO:journalctl@ceph.mon.c.smithi087.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T15:38:44.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-04T15:38:44.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: using public_addr v2:172.21.15.87:0/0 -> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-03-04T15:38:44.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: starting mon.c rank -1 at public addrs [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] at bind addrs [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:38:44.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: mon.c@-1(???) e0 preinit fsid 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:38:44.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: mon.c@-1(synchronizing).mds e1 new map 2024-03-04T15:38:44.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-04T15:38:44.301 INFO:journalctl@ceph.mon.c.smithi087.stdout: e1 2024-03-04T15:38:44.301 INFO:journalctl@ceph.mon.c.smithi087.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T15:38:44.301 INFO:journalctl@ceph.mon.c.smithi087.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T15:38:44.301 INFO:journalctl@ceph.mon.c.smithi087.stdout: legacy client fscid: -1 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout: 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout: No filesystems configured 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: Updating smithi080:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: Deploying daemon agent.smithi080 on smithi080 2024-03-04T15:38:44.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:38:44.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:44.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:44.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:44.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:44.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:44.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:44.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:44.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:44.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi087", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:38:44.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: Deploying cephadm binary to smithi087 2024-03-04T15:38:44.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:44.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:44.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: Added host smithi087 2024-03-04T15:38:44.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:44.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi087", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:38:44.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:44.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:44.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: Updating smithi087:/etc/ceph/ceph.conf 2024-03-04T15:38:44.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: Updating smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:38:44.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:44.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi087", "caps": []}]: dispatch 2024-03-04T15:38:44.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi087", "caps": []}]': finished 2024-03-04T15:38:44.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:38:44.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: Updating smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:38:44.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: Deploying daemon agent.smithi087 on smithi087 2024-03-04T15:38:44.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:38:44.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:44.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:44.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:44.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:44.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:44.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:44.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:44.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/4051206117' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/4051206117' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi005:172.21.15.5=a;smithi080:172.21.15.80=b;smithi087:172.21.15.87=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: Saving service mon spec with placement smithi005:172.21.15.5=a;smithi080:172.21.15.80=b;smithi087:172.21.15.87=c;count:3 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:44.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:44.308 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:44.308 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:38:44.308 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:44.308 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: Deploying daemon mon.c on smithi087 2024-03-04T15:38:44.308 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:44.308 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:44.308 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:44.308 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: from='client.? 172.21.15.87:0/2972819078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:38:44.308 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 ceph-mon[27664]: mon.c@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-04T15:38:44.308 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 bash[27627]: 22502b7796bd1d7de21140b603c97f5dba36dbf115b0db40a94a13df392113a4 2024-03-04T15:38:44.308 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:43 smithi087 systemd[1]: Started Ceph mon.c for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:38:44.455 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T15:38:44.456 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:38:44.687 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:38:49.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:38:49.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:49.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: mon.a calling monitor election 2024-03-04T15:38:49.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:49.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:49.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:49.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: mon.c calling monitor election 2024-03-04T15:38:49.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:49.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:49.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:49.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:49.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:49.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:38:49.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: monmap e2: 2 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],c=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:38:49.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: fsmap 2024-03-04T15:38:49.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:38:49.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: mgrmap e13: y(active, since 52s) 2024-03-04T15:38:49.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: overall HEALTH_OK 2024-03-04T15:38:49.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:38:49.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:49.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:38:49.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: mon.a calling monitor election 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: mon.c calling monitor election 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: monmap e2: 2 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],c=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: fsmap 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: mgrmap e13: y(active, since 52s) 2024-03-04T15:38:49.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: overall HEALTH_OK 2024-03-04T15:38:49.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:49.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:38:49.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:49.499 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-04T15:38:49.499 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":2,"fsid":"97c626fc-da3b-11ee-95c6-87774f69a715","modified":"2024-03-04T15:38:43.861363Z","created":"2024-03-04T15:36:59.813782Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-04T15:38:49.502 INFO:teuthology.orchestra.run.smithi087.stderr:dumped monmap epoch 2 2024-03-04T15:38:50.076 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:49 smithi087 ceph-mon[27664]: Deploying daemon mon.b on smithi080 2024-03-04T15:38:50.076 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:49 smithi087 ceph-mon[27664]: from='client.? 172.21.15.87:0/3694961280' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:38:50.076 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:49 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:50.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:49 smithi005 ceph-mon[25858]: Deploying daemon mon.b on smithi080 2024-03-04T15:38:50.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:49 smithi005 ceph-mon[25858]: from='client.? 172.21.15.87:0/3694961280' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:38:50.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:49 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:51.101 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T15:38:51.102 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:38:51.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:50 smithi087 ceph-mon[27664]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:51.342 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:38:51.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:50 smithi005 ceph-mon[25858]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:52.724 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 podman[27526]: 2024-03-04 15:38:52.602406887 +0000 UTC m=+0.127793951 container create 767a142e0f993390ba2731f21152cae5aaf79e754c49ddbd1aa52148217908ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-04T15:38:53.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 podman[27526]: 2024-03-04 15:38:52.80468938 +0000 UTC m=+0.330076411 container init 767a142e0f993390ba2731f21152cae5aaf79e754c49ddbd1aa52148217908ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2) 2024-03-04T15:38:53.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 podman[27526]: 2024-03-04 15:38:52.809710301 +0000 UTC m=+0.335097319 container start 767a142e0f993390ba2731f21152cae5aaf79e754c49ddbd1aa52148217908ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-b, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-04T15:38:53.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T15:38:53.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev), process ceph-mon, pid 7 2024-03-04T15:38:53.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pidfile_write: ignore empty --pid-file 2024-03-04T15:38:53.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: load: jerasure load: lrc 2024-03-04T15:38:53.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: RocksDB version: 7.9.2 2024-03-04T15:38:53.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Git sha 0 2024-03-04T15:38:53.098 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Compile date 2024-03-04 11:37:16 2024-03-04T15:38:53.098 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: DB SUMMARY 2024-03-04T15:38:53.098 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: DB Session ID: AVOZMXBAO91S70RMETX1 2024-03-04T15:38:53.098 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: CURRENT file: CURRENT 2024-03-04T15:38:53.098 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T15:38:53.098 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-04T15:38:53.098 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-04T15:38:53.098 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-04T15:38:53.098 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.error_if_exists: 0 2024-03-04T15:38:53.098 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.create_if_missing: 0 2024-03-04T15:38:53.098 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.paranoid_checks: 1 2024-03-04T15:38:53.099 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T15:38:53.099 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T15:38:53.099 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T15:38:53.099 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.env: 0x558ffd9cdb20 2024-03-04T15:38:53.099 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T15:38:53.099 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.info_log: 0x558fffebf8e0 2024-03-04T15:38:53.099 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T15:38:53.099 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.statistics: (nil) 2024-03-04T15:38:53.099 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.use_fsync: 0 2024-03-04T15:38:53.099 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_log_file_size: 0 2024-03-04T15:38:53.099 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T15:38:53.100 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T15:38:53.100 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T15:38:53.100 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T15:38:53.100 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.allow_fallocate: 1 2024-03-04T15:38:53.100 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T15:38:53.100 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T15:38:53.100 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.use_direct_reads: 0 2024-03-04T15:38:53.100 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T15:38:53.100 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T15:38:53.100 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.db_log_dir: 2024-03-04T15:38:53.100 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.wal_dir: 2024-03-04T15:38:53.101 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T15:38:53.101 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T15:38:53.101 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T15:38:53.101 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T15:38:53.101 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T15:38:53.101 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T15:38:53.101 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T15:38:53.101 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T15:38:53.101 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.write_buffer_manager: 0x558fffe93400 2024-03-04T15:38:53.101 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T15:38:53.101 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T15:38:53.101 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T15:38:53.101 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T15:38:53.101 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T15:38:53.102 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T15:38:53.102 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T15:38:53.102 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T15:38:53.102 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.unordered_write: 0 2024-03-04T15:38:53.102 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T15:38:53.102 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T15:38:53.102 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T15:38:53.102 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T15:38:53.102 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.row_cache: None 2024-03-04T15:38:53.102 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.wal_filter: None 2024-03-04T15:38:53.104 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T15:38:53.104 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T15:38:53.104 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.two_write_queues: 0 2024-03-04T15:38:53.104 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T15:38:53.104 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.wal_compression: 0 2024-03-04T15:38:53.105 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.atomic_flush: 0 2024-03-04T15:38:53.105 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T15:38:53.105 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T15:38:53.105 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T15:38:53.105 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.log_readahead_size: 0 2024-03-04T15:38:53.105 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T15:38:53.105 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T15:38:53.105 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T15:38:53.105 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T15:38:53.105 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T15:38:53.105 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T15:38:53.105 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T15:38:53.105 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_background_jobs: 2 2024-03-04T15:38:53.106 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_background_compactions: -1 2024-03-04T15:38:53.106 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_subcompactions: 1 2024-03-04T15:38:53.106 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T15:38:53.106 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T15:38:53.106 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T15:38:53.106 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T15:38:53.106 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T15:38:53.106 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T15:38:53.106 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T15:38:53.106 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T15:38:53.106 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_open_files: -1 2024-03-04T15:38:53.107 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T15:38:53.107 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T15:38:53.107 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T15:38:53.107 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T15:38:53.107 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_background_flushes: -1 2024-03-04T15:38:53.107 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Compression algorithms supported: 2024-03-04T15:38:53.107 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T15:38:53.107 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: kZSTD supported: 0 2024-03-04T15:38:53.107 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: kXpressCompression supported: 0 2024-03-04T15:38:53.107 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T15:38:53.107 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: kZlibCompression supported: 1 2024-03-04T15:38:53.107 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: kSnappyCompression supported: 1 2024-03-04T15:38:53.107 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: kLZ4Compression supported: 1 2024-03-04T15:38:53.108 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: kBZip2Compression supported: 0 2024-03-04T15:38:53.108 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T15:38:53.108 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T15:38:53.108 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-04T15:38:53.108 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T15:38:53.108 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T15:38:53.108 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.merge_operator: 2024-03-04T15:38:53.108 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compaction_filter: None 2024-03-04T15:38:53.108 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T15:38:53.108 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T15:38:53.108 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T15:38:53.108 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T15:38:53.109 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558fffebf9e0) 2024-03-04T15:38:53.109 INFO:journalctl@ceph.mon.b.smithi080.stdout: cache_index_and_filter_blocks: 1 2024-03-04T15:38:53.109 INFO:journalctl@ceph.mon.b.smithi080.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T15:38:53.109 INFO:journalctl@ceph.mon.b.smithi080.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T15:38:53.109 INFO:journalctl@ceph.mon.b.smithi080.stdout: pin_top_level_index_and_filter: 1 2024-03-04T15:38:53.109 INFO:journalctl@ceph.mon.b.smithi080.stdout: index_type: 0 2024-03-04T15:38:53.109 INFO:journalctl@ceph.mon.b.smithi080.stdout: data_block_index_type: 0 2024-03-04T15:38:53.109 INFO:journalctl@ceph.mon.b.smithi080.stdout: index_shortening: 1 2024-03-04T15:38:53.109 INFO:journalctl@ceph.mon.b.smithi080.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T15:38:53.109 INFO:journalctl@ceph.mon.b.smithi080.stdout: checksum: 4 2024-03-04T15:38:53.109 INFO:journalctl@ceph.mon.b.smithi080.stdout: no_block_cache: 0 2024-03-04T15:38:53.109 INFO:journalctl@ceph.mon.b.smithi080.stdout: block_cache: 0x558fffec5090 2024-03-04T15:38:53.109 INFO:journalctl@ceph.mon.b.smithi080.stdout: block_cache_name: BinnedLRUCache 2024-03-04T15:38:53.109 INFO:journalctl@ceph.mon.b.smithi080.stdout: block_cache_options: 2024-03-04T15:38:53.109 INFO:journalctl@ceph.mon.b.smithi080.stdout: capacity : 536870912 2024-03-04T15:38:53.110 INFO:journalctl@ceph.mon.b.smithi080.stdout: num_shard_bits : 4 2024-03-04T15:38:53.110 INFO:journalctl@ceph.mon.b.smithi080.stdout: strict_capacity_limit : 0 2024-03-04T15:38:53.110 INFO:journalctl@ceph.mon.b.smithi080.stdout: high_pri_pool_ratio: 0.000 2024-03-04T15:38:53.110 INFO:journalctl@ceph.mon.b.smithi080.stdout: block_cache_compressed: (nil) 2024-03-04T15:38:53.110 INFO:journalctl@ceph.mon.b.smithi080.stdout: persistent_cache: (nil) 2024-03-04T15:38:53.110 INFO:journalctl@ceph.mon.b.smithi080.stdout: block_size: 4096 2024-03-04T15:38:53.110 INFO:journalctl@ceph.mon.b.smithi080.stdout: block_size_deviation: 10 2024-03-04T15:38:53.110 INFO:journalctl@ceph.mon.b.smithi080.stdout: block_restart_interval: 16 2024-03-04T15:38:53.110 INFO:journalctl@ceph.mon.b.smithi080.stdout: index_block_restart_interval: 1 2024-03-04T15:38:53.110 INFO:journalctl@ceph.mon.b.smithi080.stdout: metadata_block_size: 4096 2024-03-04T15:38:53.110 INFO:journalctl@ceph.mon.b.smithi080.stdout: partition_filters: 0 2024-03-04T15:38:53.110 INFO:journalctl@ceph.mon.b.smithi080.stdout: use_delta_encoding: 1 2024-03-04T15:38:53.110 INFO:journalctl@ceph.mon.b.smithi080.stdout: filter_policy: bloomfilter 2024-03-04T15:38:53.110 INFO:journalctl@ceph.mon.b.smithi080.stdout: whole_key_filtering: 1 2024-03-04T15:38:53.111 INFO:journalctl@ceph.mon.b.smithi080.stdout: verify_compression: 0 2024-03-04T15:38:53.111 INFO:journalctl@ceph.mon.b.smithi080.stdout: read_amp_bytes_per_bit: 0 2024-03-04T15:38:53.111 INFO:journalctl@ceph.mon.b.smithi080.stdout: format_version: 5 2024-03-04T15:38:53.111 INFO:journalctl@ceph.mon.b.smithi080.stdout: enable_index_compression: 1 2024-03-04T15:38:53.111 INFO:journalctl@ceph.mon.b.smithi080.stdout: block_align: 0 2024-03-04T15:38:53.111 INFO:journalctl@ceph.mon.b.smithi080.stdout: max_auto_readahead_size: 262144 2024-03-04T15:38:53.111 INFO:journalctl@ceph.mon.b.smithi080.stdout: prepopulate_block_cache: 0 2024-03-04T15:38:53.111 INFO:journalctl@ceph.mon.b.smithi080.stdout: initial_auto_readahead_size: 8192 2024-03-04T15:38:53.111 INFO:journalctl@ceph.mon.b.smithi080.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T15:38:53.111 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T15:38:53.111 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T15:38:53.111 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compression: NoCompression 2024-03-04T15:38:53.111 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T15:38:53.112 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T15:38:53.112 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T15:38:53.112 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.num_levels: 7 2024-03-04T15:38:53.112 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T15:38:53.112 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T15:38:53.112 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T15:38:53.112 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T15:38:53.112 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T15:38:53.112 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T15:38:53.112 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T15:38:53.112 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:38:53.112 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T15:38:53.113 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T15:38:53.113 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:38:53.113 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:38:53.113 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T15:38:53.113 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T15:38:53.113 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T15:38:53.113 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T15:38:53.113 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:38:53.113 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:38:53.113 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T15:38:53.113 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T15:38:53.113 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:38:53.113 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T15:38:53.114 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T15:38:53.114 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T15:38:53.114 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T15:38:53.114 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T15:38:53.114 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T15:38:53.114 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T15:38:53.114 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T15:38:53.114 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T15:38:53.114 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T15:38:53.114 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T15:38:53.114 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T15:38:53.114 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T15:38:53.114 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T15:38:53.115 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T15:38:53.115 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T15:38:53.115 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T15:38:53.115 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T15:38:53.115 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T15:38:53.115 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T15:38:53.115 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T15:38:53.115 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T15:38:53.115 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T15:38:53.115 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T15:38:53.115 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T15:38:53.115 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T15:38:53.115 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T15:38:53.115 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T15:38:53.116 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T15:38:53.116 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T15:38:53.116 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T15:38:53.116 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T15:38:53.116 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.table_properties_collectors: 2024-03-04T15:38:53.116 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.inplace_update_support: 0 2024-03-04T15:38:53.116 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T15:38:53.116 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T15:38:53.116 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T15:38:53.116 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T15:38:53.116 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.bloom_locality: 0 2024-03-04T15:38:53.116 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.max_successive_merges: 0 2024-03-04T15:38:53.117 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T15:38:53.117 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T15:38:53.117 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T15:38:53.117 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T15:38:53.117 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.ttl: 2592000 2024-03-04T15:38:53.117 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T15:38:53.117 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T15:38:53.117 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T15:38:53.117 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.enable_blob_files: false 2024-03-04T15:38:53.117 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.min_blob_size: 0 2024-03-04T15:38:53.117 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T15:38:53.117 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T15:38:53.117 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T15:38:53.117 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T15:38:53.117 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T15:38:53.118 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T15:38:53.118 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T15:38:53.118 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T15:38:53.118 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-04T15:38:53.118 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-04T15:38:53.118 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 833f7791-012f-4143-b5e0-0a11dd3297e6 2024-03-04T15:38:53.118 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709566732839950, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-04T15:38:53.118 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-04T15:38:53.118 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709566732840355, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709566732, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "833f7791-012f-4143-b5e0-0a11dd3297e6", "db_session_id": "AVOZMXBAO91S70RMETX1", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-04T15:38:53.118 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709566732840433, "job": 1, "event": "recovery_finished"} 2024-03-04T15:38:53.118 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-04T15:38:53.118 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T15:38:53.118 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558ffffb8000 2024-03-04T15:38:53.118 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: DB pointer 0x558ffffa2000 2024-03-04T15:38:53.119 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T15:38:53.119 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T15:38:53.119 INFO:journalctl@ceph.mon.b.smithi080.stdout: ** DB Stats ** 2024-03-04T15:38:53.119 INFO:journalctl@ceph.mon.b.smithi080.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:38:53.119 INFO:journalctl@ceph.mon.b.smithi080.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T15:38:53.119 INFO:journalctl@ceph.mon.b.smithi080.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:38:53.119 INFO:journalctl@ceph.mon.b.smithi080.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:38:53.119 INFO:journalctl@ceph.mon.b.smithi080.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T15:38:53.119 INFO:journalctl@ceph.mon.b.smithi080.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:38:53.119 INFO:journalctl@ceph.mon.b.smithi080.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:38:53.119 INFO:journalctl@ceph.mon.b.smithi080.stdout: 2024-03-04T15:38:53.119 INFO:journalctl@ceph.mon.b.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-04T15:38:53.119 INFO:journalctl@ceph.mon.b.smithi080.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:38:53.119 INFO:journalctl@ceph.mon.b.smithi080.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T15:38:53.119 INFO:journalctl@ceph.mon.b.smithi080.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:38:53.120 INFO:journalctl@ceph.mon.b.smithi080.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:38:53.120 INFO:journalctl@ceph.mon.b.smithi080.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:38:53.120 INFO:journalctl@ceph.mon.b.smithi080.stdout: 2024-03-04T15:38:53.120 INFO:journalctl@ceph.mon.b.smithi080.stdout: ** Compaction Stats [default] ** 2024-03-04T15:38:53.120 INFO:journalctl@ceph.mon.b.smithi080.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:38:53.120 INFO:journalctl@ceph.mon.b.smithi080.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:38:53.120 INFO:journalctl@ceph.mon.b.smithi080.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:38:53.120 INFO:journalctl@ceph.mon.b.smithi080.stdout: 2024-03-04T15:38:53.120 INFO:journalctl@ceph.mon.b.smithi080.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T15:38:53.120 INFO:journalctl@ceph.mon.b.smithi080.stdout: 2024-03-04T15:38:53.120 INFO:journalctl@ceph.mon.b.smithi080.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:38:53.120 INFO:journalctl@ceph.mon.b.smithi080.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T15:38:53.120 INFO:journalctl@ceph.mon.b.smithi080.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T15:38:53.121 INFO:journalctl@ceph.mon.b.smithi080.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T15:38:53.121 INFO:journalctl@ceph.mon.b.smithi080.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T15:38:53.121 INFO:journalctl@ceph.mon.b.smithi080.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T15:38:53.121 INFO:journalctl@ceph.mon.b.smithi080.stdout: Cumulative compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:38:53.121 INFO:journalctl@ceph.mon.b.smithi080.stdout: Interval compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:38:53.121 INFO:journalctl@ceph.mon.b.smithi080.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T15:38:53.121 INFO:journalctl@ceph.mon.b.smithi080.stdout: Block cache BinnedLRUCache@0x558fffec5090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-04T15:38:53.121 INFO:journalctl@ceph.mon.b.smithi080.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-04T15:38:53.121 INFO:journalctl@ceph.mon.b.smithi080.stdout: 2024-03-04T15:38:53.121 INFO:journalctl@ceph.mon.b.smithi080.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T15:38:53.121 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-04T15:38:53.122 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: using public_addr v2:172.21.15.80:0/0 -> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] 2024-03-04T15:38:53.122 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: starting mon.b rank -1 at public addrs [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] at bind addrs [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:38:53.122 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.b@-1(???) e0 preinit fsid 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:38:53.122 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.b@-1(synchronizing).mds e1 new map 2024-03-04T15:38:53.122 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-04T15:38:53.122 INFO:journalctl@ceph.mon.b.smithi080.stdout: e1 2024-03-04T15:38:53.122 INFO:journalctl@ceph.mon.b.smithi080.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T15:38:53.122 INFO:journalctl@ceph.mon.b.smithi080.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T15:38:53.122 INFO:journalctl@ceph.mon.b.smithi080.stdout: legacy client fscid: -1 2024-03-04T15:38:53.122 INFO:journalctl@ceph.mon.b.smithi080.stdout: 2024-03-04T15:38:53.122 INFO:journalctl@ceph.mon.b.smithi080.stdout: No filesystems configured 2024-03-04T15:38:53.122 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-04T15:38:53.123 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-04T15:38:53.123 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-04T15:38:53.123 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-04T15:38:53.123 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-04T15:38:53.123 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-04T15:38:53.123 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-04T15:38:53.123 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:38:53.123 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:38:53.123 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:38:53.123 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:38:53.123 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: Updating smithi080:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:38:53.123 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: Deploying daemon agent.smithi080 on smithi080 2024-03-04T15:38:53.123 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:38:53.123 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.124 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.124 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.124 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:53.124 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:53.124 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:53.124 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.124 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.124 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:53.124 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:53.124 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:53.124 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.124 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.125 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.125 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.125 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.125 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi087", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:38:53.125 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: Deploying cephadm binary to smithi087 2024-03-04T15:38:53.125 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.125 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.125 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.125 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.125 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.125 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.125 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.125 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.125 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.126 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: Added host smithi087 2024-03-04T15:38:53.126 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:53.126 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.126 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.126 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.126 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi087", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:38:53.126 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:53.126 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:53.126 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: Updating smithi087:/etc/ceph/ceph.conf 2024-03-04T15:38:53.126 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: Updating smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:38:53.126 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.126 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.126 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.126 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi087", "caps": []}]: dispatch 2024-03-04T15:38:53.127 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi087", "caps": []}]': finished 2024-03-04T15:38:53.127 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:38:53.127 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: Updating smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:38:53.127 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: Deploying daemon agent.smithi087 on smithi087 2024-03-04T15:38:53.127 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:38:53.127 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.127 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.127 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.127 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.127 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:53.127 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:53.127 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:53.127 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.127 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:53.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:53.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:53.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/4051206117' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-04T15:38:53.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/4051206117' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-04T15:38:53.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:38:53.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.128 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi005:172.21.15.5=a;smithi080:172.21.15.80=b;smithi087:172.21.15.87=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:38:53.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: Saving service mon spec with placement smithi005:172.21.15.5=a;smithi080:172.21.15.80=b;smithi087:172.21.15.87=c;count:3 2024-03-04T15:38:53.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:53.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:53.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:53.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:38:53.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:53.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: Deploying daemon mon.c on smithi087 2024-03-04T15:38:53.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.129 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.130 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.130 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='client.? 172.21.15.87:0/2972819078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:38:53.130 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:38:53.130 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:53.130 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.a calling monitor election 2024-03-04T15:38:53.130 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.130 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:53.130 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:53.130 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.c calling monitor election 2024-03-04T15:38:53.130 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.130 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:53.130 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:53.130 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.130 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:53.130 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:38:53.131 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: monmap e2: 2 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],c=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:38:53.131 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: fsmap 2024-03-04T15:38:53.131 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:38:53.131 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mgrmap e13: y(active, since 52s) 2024-03-04T15:38:53.131 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: overall HEALTH_OK 2024-03-04T15:38:53.131 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.131 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.131 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.131 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.131 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.131 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.131 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.131 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.131 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:53.132 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:38:53.132 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:53.132 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: Deploying daemon mon.b on smithi080 2024-03-04T15:38:53.132 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='client.? 172.21.15.87:0/3694961280' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:38:53.132 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:53.132 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.132 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.132 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 ceph-mon[27564]: mon.b@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-04T15:38:53.132 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 bash[27526]: 767a142e0f993390ba2731f21152cae5aaf79e754c49ddbd1aa52148217908ad 2024-03-04T15:38:53.132 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:52 smithi080 systemd[1]: Started Ceph mon.b for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:38:53.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:52 smithi005 ceph-mon[25858]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:53.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:52 smithi087 ceph-mon[27664]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:57.897 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-03-04T15:38:57.897 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":3,"fsid":"97c626fc-da3b-11ee-95c6-87774f69a715","modified":"2024-03-04T15:38:52.871004Z","created":"2024-03-04T15:36:59.813782Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:3300","nonce":0},{"type":"v1","addr":"172.21.15.80:6789","nonce":0}]},"addr":"172.21.15.80:6789/0","public_addr":"172.21.15.80:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-04T15:38:57.898 INFO:teuthology.orchestra.run.smithi087.stderr:dumped monmap epoch 3 2024-03-04T15:38:58.234 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:38:58.234 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: mon.a calling monitor election 2024-03-04T15:38:58.234 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: mon.c calling monitor election 2024-03-04T15:38:58.234 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:38:58.234 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:58.234 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='client.? 172.21.15.87:0/4106298355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:38:58.234 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:38:58.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:58.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:38:58.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:38:58.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:58.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:38:58.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:38:58.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:38:58.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],c=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:38:58.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: fsmap 2024-03-04T15:38:58.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:38:58.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: mgrmap e13: y(active, since 61s) 2024-03-04T15:38:58.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T15:38:58.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T15:38:58.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T15:38:58.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: mon.b (rank 2) addr [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] is down (out of quorum) 2024-03-04T15:38:58.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:58.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:58.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:58.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:58.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:58.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:58.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:58.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:38:58.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: mon.a calling monitor election 2024-03-04T15:38:58.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: mon.c calling monitor election 2024-03-04T15:38:58.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:38:58.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:38:58.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='client.? 172.21.15.87:0/4106298355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:38:58.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:38:58.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:58.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:38:58.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:38:58.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:58.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:38:58.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:38:58.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:38:58.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],c=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:38:58.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: fsmap 2024-03-04T15:38:58.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:38:58.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: mgrmap e13: y(active, since 61s) 2024-03-04T15:38:58.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T15:38:58.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T15:38:58.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T15:38:58.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: mon.b (rank 2) addr [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] is down (out of quorum) 2024-03-04T15:38:58.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:58.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:58.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:58.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:58.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:58.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:58.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:58.606 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-04T15:38:58.606 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph config generate-minimal-conf 2024-03-04T15:38:58.867 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:38:59.231 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:59.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:59.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:59.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-04T15:38:59.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-04T15:38:59.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: Updating smithi087:/etc/ceph/ceph.conf 2024-03-04T15:38:59.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: Updating smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:38:59.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: Updating smithi080:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:38:59.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: Updating smithi005:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:38:59.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:59.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:59.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:59.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:59.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:59.233 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:59.233 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:59.233 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:38:59.233 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:59.233 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:38:59.233 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:38:59.233 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:38:59.233 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:38:58 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:59.352 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:38:59.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:38:59.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:38:59.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-04T15:38:59.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-04T15:38:59.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: Updating smithi087:/etc/ceph/ceph.conf 2024-03-04T15:38:59.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: Updating smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:38:59.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: Updating smithi080:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:38:59.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: Updating smithi005:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:38:59.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:59.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:59.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:59.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:59.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:59.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:59.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:38:59.354 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:38:59.354 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:38:59.354 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:38:59.354 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:38:59.354 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:38:59.354 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:38:58 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:00.062 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:39:00.062 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: mon.a calling monitor election 2024-03-04T15:39:00.062 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: mon.c calling monitor election 2024-03-04T15:39:00.062 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:39:00.062 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:39:00.062 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='client.? 172.21.15.87:0/4106298355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:39:00.062 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:39:00.062 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:00.062 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:39:00.062 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:39:00.062 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:00.062 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:39:00.063 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:39:00.063 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:39:00.063 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],c=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:39:00.063 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: fsmap 2024-03-04T15:39:00.063 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:39:00.063 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: mgrmap e13: y(active, since 61s) 2024-03-04T15:39:00.063 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T15:39:00.063 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T15:39:00.063 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T15:39:00.063 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: mon.b (rank 2) addr [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] is down (out of quorum) 2024-03-04T15:39:00.063 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:00.063 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:00.063 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:00.063 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:00.063 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:00.064 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:00.064 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:00.064 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:00.064 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:00.064 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:00.064 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-04T15:39:00.064 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-04T15:39:00.064 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: Updating smithi087:/etc/ceph/ceph.conf 2024-03-04T15:39:00.064 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: Updating smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:39:00.064 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: Updating smithi080:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:39:00.064 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: Updating smithi005:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:39:00.064 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:00.064 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:00.064 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:00.064 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:00.065 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:00.065 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:00.065 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:00.065 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:39:00.065 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:00.065 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:39:00.065 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:39:00.065 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:39:00.065 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:38:59 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:00.895 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:00 smithi087 ceph-mon[27664]: mon.b calling monitor election 2024-03-04T15:39:00.895 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:00 smithi087 ceph-mon[27664]: mon.b calling monitor election 2024-03-04T15:39:00.896 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:00 smithi087 ceph-mon[27664]: mon.a calling monitor election 2024-03-04T15:39:00.896 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:00 smithi087 ceph-mon[27664]: mon.c calling monitor election 2024-03-04T15:39:00.896 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:00 smithi087 ceph-mon[27664]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T15:39:00.896 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:00 smithi087 ceph-mon[27664]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],c=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:39:00.896 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:00 smithi087 ceph-mon[27664]: fsmap 2024-03-04T15:39:00.896 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:00 smithi087 ceph-mon[27664]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:39:00.896 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:00 smithi087 ceph-mon[27664]: mgrmap e13: y(active, since 63s) 2024-03-04T15:39:00.896 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:00 smithi087 ceph-mon[27664]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T15:39:00.896 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:00 smithi087 ceph-mon[27664]: Cluster is now healthy 2024-03-04T15:39:00.896 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:00 smithi087 ceph-mon[27664]: overall HEALTH_OK 2024-03-04T15:39:01.014 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:00 smithi005 ceph-mon[25858]: mon.b calling monitor election 2024-03-04T15:39:01.014 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:00 smithi005 ceph-mon[25858]: mon.b calling monitor election 2024-03-04T15:39:01.014 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:00 smithi005 ceph-mon[25858]: mon.a calling monitor election 2024-03-04T15:39:01.014 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:00 smithi005 ceph-mon[25858]: mon.c calling monitor election 2024-03-04T15:39:01.014 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:00 smithi005 ceph-mon[25858]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T15:39:01.014 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:00 smithi005 ceph-mon[25858]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],c=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:39:01.014 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:00 smithi005 ceph-mon[25858]: fsmap 2024-03-04T15:39:01.014 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:00 smithi005 ceph-mon[25858]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:39:01.014 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:00 smithi005 ceph-mon[25858]: mgrmap e13: y(active, since 63s) 2024-03-04T15:39:01.014 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:00 smithi005 ceph-mon[25858]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T15:39:01.014 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:00 smithi005 ceph-mon[25858]: Cluster is now healthy 2024-03-04T15:39:01.014 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:00 smithi005 ceph-mon[25858]: overall HEALTH_OK 2024-03-04T15:39:01.077 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:00 smithi080 ceph-mon[27564]: mon.b calling monitor election 2024-03-04T15:39:01.077 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:00 smithi080 ceph-mon[27564]: mon.b calling monitor election 2024-03-04T15:39:01.077 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:00 smithi080 ceph-mon[27564]: mon.a calling monitor election 2024-03-04T15:39:01.077 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:00 smithi080 ceph-mon[27564]: mon.c calling monitor election 2024-03-04T15:39:01.077 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:00 smithi080 ceph-mon[27564]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T15:39:01.077 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:00 smithi080 ceph-mon[27564]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0],c=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:39:01.077 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:00 smithi080 ceph-mon[27564]: fsmap 2024-03-04T15:39:01.077 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:00 smithi080 ceph-mon[27564]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:39:01.077 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:00 smithi080 ceph-mon[27564]: mgrmap e13: y(active, since 63s) 2024-03-04T15:39:01.077 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:00 smithi080 ceph-mon[27564]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T15:39:01.077 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:00 smithi080 ceph-mon[27564]: Cluster is now healthy 2024-03-04T15:39:01.077 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:00 smithi080 ceph-mon[27564]: overall HEALTH_OK 2024-03-04T15:39:01.779 INFO:teuthology.orchestra.run.smithi005.stdout:# minimal ceph.conf for 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:39:01.779 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2024-03-04T15:39:01.779 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:39:01.780 INFO:teuthology.orchestra.run.smithi005.stdout: mon_host = [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] 2024-03-04T15:39:01.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:01 smithi005 ceph-mon[25858]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:01.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:01 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:39:01.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:01 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:01.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:01 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:01.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:01 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:01.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:01 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:01.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:01 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:01.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:01 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:39:01.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:01 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:39:01.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:01 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:01.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:01 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:01.988 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:01 smithi080 ceph-mon[27564]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:01.988 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:01 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:39:01.988 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:01 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:01.988 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:01 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:01.988 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:01 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:01.988 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:01 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:01.988 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:01 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:01.988 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:01 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:39:01.988 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:01 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:39:01.988 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:01 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:01.988 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:01 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:02.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:01 smithi087 ceph-mon[27664]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:02.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:01 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:39:02.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:01 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:02.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:01 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:02.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:01 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:02.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:01 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:02.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:01 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:02.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:01 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:39:02.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:01 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:39:02.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:01 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:02.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:01 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:02.311 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-04T15:39:02.311 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-04T15:39:02.311 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T15:39:02.340 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-04T15:39:02.340 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:39:02.407 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-04T15:39:02.407 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T15:39:02.438 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-04T15:39:02.438 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:39:02.504 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-04T15:39:02.504 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T15:39:02.540 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-04T15:39:02.540 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:39:02.610 INFO:tasks.cephadm:Adding mgr.y on smithi005 2024-03-04T15:39:02.610 INFO:tasks.cephadm:Adding mgr.x on smithi080 2024-03-04T15:39:02.610 INFO:tasks.cephadm:Adding mgr.z on smithi087 2024-03-04T15:39:02.610 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph orch apply mgr '3;smithi005=y;smithi080=x;smithi087=z' 2024-03-04T15:39:02.886 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:39:03.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:02 smithi087 ceph-mon[27664]: Reconfiguring mon.b (monmap changed)... 2024-03-04T15:39:03.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:02 smithi087 ceph-mon[27664]: Reconfiguring daemon mon.b on smithi080 2024-03-04T15:39:03.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:02 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/2098708806' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:03.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:02 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:03.008 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:02 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:03.008 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:02 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:03.008 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:02 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:03.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:02 smithi080 ceph-mon[27564]: Reconfiguring mon.b (monmap changed)... 2024-03-04T15:39:03.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:02 smithi080 ceph-mon[27564]: Reconfiguring daemon mon.b on smithi080 2024-03-04T15:39:03.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:02 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/2098708806' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:03.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:02 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:03.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:02 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:03.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:02 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:03.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:02 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:03.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:02 smithi005 ceph-mon[25858]: Reconfiguring mon.b (monmap changed)... 2024-03-04T15:39:03.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:02 smithi005 ceph-mon[25858]: Reconfiguring daemon mon.b on smithi080 2024-03-04T15:39:03.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:02 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2098708806' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:03.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:02 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:03.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:02 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:03.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:02 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:03.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:02 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:04.003 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:03 smithi087 ceph-mon[27664]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:04.004 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:03 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:04.004 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:03 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:04.004 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:03 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:39:04.004 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:03 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:39:04.004 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:03 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:04.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:03 smithi080 ceph-mon[27564]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:04.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:03 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:04.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:03 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:04.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:03 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:39:04.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:03 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:39:04.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:03 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:04.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:03 smithi005 ceph-mon[25858]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:04.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:03 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:04.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:03 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:04.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:03 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:39:04.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:03 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:39:04.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:03 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:05.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:04 smithi087 ceph-mon[27664]: Reconfiguring mon.c (monmap changed)... 2024-03-04T15:39:05.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:04 smithi087 ceph-mon[27664]: Reconfiguring daemon mon.c on smithi087 2024-03-04T15:39:05.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:04 smithi080 ceph-mon[27564]: Reconfiguring mon.c (monmap changed)... 2024-03-04T15:39:05.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:04 smithi080 ceph-mon[27564]: Reconfiguring daemon mon.c on smithi087 2024-03-04T15:39:05.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:04 smithi005 ceph-mon[25858]: Reconfiguring mon.c (monmap changed)... 2024-03-04T15:39:05.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:04 smithi005 ceph-mon[25858]: Reconfiguring daemon mon.c on smithi087 2024-03-04T15:39:05.732 INFO:teuthology.orchestra.run.smithi087.stdout:Scheduled mgr update... 2024-03-04T15:39:05.979 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:05 smithi087 ceph-mon[27664]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:05.979 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:05 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:05.979 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:05 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:05.979 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:05 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:05.979 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:05 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:05.979 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:05 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:05.979 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:05 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:06.006 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:05 smithi005 ceph-mon[25858]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:06.007 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:05 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:06.007 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:05 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:06.007 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:05 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:06.007 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:05 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:06.007 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:05 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:06.007 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:05 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:06.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:05 smithi080 ceph-mon[27564]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:06.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:05 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:06.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:05 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:06.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:05 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:06.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:05 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:06.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:05 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:06.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:05 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:06.318 DEBUG:teuthology.orchestra.run.smithi080:mgr.x> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mgr.x.service 2024-03-04T15:39:06.321 DEBUG:teuthology.orchestra.run.smithi087:mgr.z> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mgr.z.service 2024-03-04T15:39:06.323 INFO:tasks.cephadm:Deploying OSDs... 2024-03-04T15:39:06.323 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-04T15:39:06.323 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:39:06.341 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-04T15:39:06.341 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/nvme1n1 2024-03-04T15:39:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/nvme1n1 2024-03-04T15:39:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:39:06.397 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 764 Links: 1 Device type: 103,2 2024-03-04T15:39:06.397 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:39:06.397 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:39:06.397 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-04 15:38:48.076599099 +0000 2024-03-04T15:39:06.397 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-04 15:26:33.856542129 +0000 2024-03-04T15:39:06.397 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-04 15:26:33.856542129 +0000 2024-03-04T15:39:06.397 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-04 15:26:33.849542260 +0000 2024-03-04T15:39:06.397 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-04T15:39:06.466 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-04T15:39:06.467 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-04T15:39:06.467 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000124768 s, 4.1 MB/s 2024-03-04T15:39:06.468 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-04T15:39:06.524 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/nvme2n1 2024-03-04T15:39:06.580 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/nvme2n1 2024-03-04T15:39:06.580 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:39:06.580 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 776 Links: 1 Device type: 103,4 2024-03-04T15:39:06.581 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:39:06.581 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:39:06.581 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-04 15:38:48.084598938 +0000 2024-03-04T15:39:06.581 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-04 15:26:34.015539165 +0000 2024-03-04T15:39:06.581 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-04 15:26:34.015539165 +0000 2024-03-04T15:39:06.581 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-04 15:26:34.009539277 +0000 2024-03-04T15:39:06.581 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-04T15:39:06.647 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-04T15:39:06.647 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-04T15:39:06.647 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000131146 s, 3.9 MB/s 2024-03-04T15:39:06.648 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-04T15:39:06.704 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/nvme3n1 2024-03-04T15:39:06.760 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/nvme3n1 2024-03-04T15:39:06.760 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:39:06.760 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-03-04T15:39:06.760 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:39:06.760 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:39:06.760 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-04 15:38:48.093598756 +0000 2024-03-04T15:39:06.760 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-04 15:26:34.165536368 +0000 2024-03-04T15:39:06.760 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-04 15:26:34.165536368 +0000 2024-03-04T15:39:06.760 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-04 15:26:34.155536555 +0000 2024-03-04T15:39:06.760 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-04T15:39:06.825 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-04T15:39:06.826 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-04T15:39:06.826 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000107108 s, 4.8 MB/s 2024-03-04T15:39:06.827 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-04T15:39:06.882 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/nvme4n1 2024-03-04T15:39:06.939 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/nvme4n1 2024-03-04T15:39:06.939 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:39:06.939 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-03-04T15:39:06.939 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:39:06.939 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:39:06.939 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-04 15:38:48.101598595 +0000 2024-03-04T15:39:06.939 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-04 15:26:34.311533647 +0000 2024-03-04T15:39:06.939 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-04 15:26:34.311533647 +0000 2024-03-04T15:39:06.939 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-04 15:26:34.301533833 +0000 2024-03-04T15:39:06.940 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-04T15:39:06.996 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:06 smithi005 ceph-mon[25858]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi005=y;smithi080=x;smithi087=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:39:06.996 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:06 smithi005 ceph-mon[25858]: Saving service mgr spec with placement smithi005=y;smithi080=x;smithi087=z;count:3 2024-03-04T15:39:06.996 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:06.997 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:06.997 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:06.997 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:06.997 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:06.997 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:39:06.997 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:39:06.997 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:39:06.997 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:06.997 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:06 smithi005 ceph-mon[25858]: Deploying daemon mgr.z on smithi087 2024-03-04T15:39:07.008 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-04T15:39:07.008 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-04T15:39:07.008 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000115467 s, 4.4 MB/s 2024-03-04T15:39:07.009 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-04T15:39:07.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:06 smithi087 ceph-mon[27664]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi005=y;smithi080=x;smithi087=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:39:07.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:06 smithi087 ceph-mon[27664]: Saving service mgr spec with placement smithi005=y;smithi080=x;smithi087=z;count:3 2024-03-04T15:39:07.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:07.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:07.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:07.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:07.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:07.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:39:07.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:39:07.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:39:07.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:07.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:06 smithi087 ceph-mon[27664]: Deploying daemon mgr.z on smithi087 2024-03-04T15:39:07.065 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-04T15:39:07.065 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:39:07.083 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-04T15:39:07.084 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/nvme1n1 2024-03-04T15:39:07.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:06 smithi080 ceph-mon[27564]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi005=y;smithi080=x;smithi087=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:39:07.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:06 smithi080 ceph-mon[27564]: Saving service mgr spec with placement smithi005=y;smithi080=x;smithi087=z;count:3 2024-03-04T15:39:07.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:07.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:07.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:07.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:07.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:07.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:39:07.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:39:07.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:39:07.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:07.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:06 smithi080 ceph-mon[27564]: Deploying daemon mgr.z on smithi087 2024-03-04T15:39:07.114 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/nvme1n1 2024-03-04T15:39:07.114 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:39:07.114 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-04T15:39:07.114 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:39:07.114 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:39:07.114 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-04 15:39:00.644602167 +0000 2024-03-04T15:39:07.114 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-04 15:26:35.487719719 +0000 2024-03-04T15:39:07.114 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-04 15:26:35.487719719 +0000 2024-03-04T15:39:07.114 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-04 15:26:35.475719977 +0000 2024-03-04T15:39:07.114 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-04T15:39:07.183 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-04T15:39:07.184 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-04T15:39:07.184 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000291834 s, 1.8 MB/s 2024-03-04T15:39:07.185 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-04T15:39:07.242 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/nvme2n1 2024-03-04T15:39:07.299 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/nvme2n1 2024-03-04T15:39:07.299 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:39:07.299 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-04T15:39:07.299 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:39:07.299 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:39:07.299 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-04 15:39:00.655601920 +0000 2024-03-04T15:39:07.299 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-04 15:26:35.631716612 +0000 2024-03-04T15:39:07.299 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-04 15:26:35.631716612 +0000 2024-03-04T15:39:07.299 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-04 15:26:35.620716849 +0000 2024-03-04T15:39:07.299 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-04T15:39:07.366 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-04T15:39:07.366 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-04T15:39:07.366 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000259244 s, 2.0 MB/s 2024-03-04T15:39:07.368 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-04T15:39:07.424 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/nvme3n1 2024-03-04T15:39:07.480 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/nvme3n1 2024-03-04T15:39:07.480 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:39:07.480 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-04T15:39:07.480 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:39:07.480 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:39:07.481 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-04 15:39:00.667601652 +0000 2024-03-04T15:39:07.481 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-04 15:26:35.783713332 +0000 2024-03-04T15:39:07.481 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-04 15:26:35.783713332 +0000 2024-03-04T15:39:07.481 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-04 15:26:35.775713505 +0000 2024-03-04T15:39:07.481 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-04T15:39:07.548 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-04T15:39:07.548 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-04T15:39:07.548 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000211854 s, 2.4 MB/s 2024-03-04T15:39:07.549 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-04T15:39:07.605 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/nvme4n1 2024-03-04T15:39:07.659 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/nvme4n1 2024-03-04T15:39:07.660 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:39:07.660 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-04T15:39:07.660 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:39:07.660 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:39:07.660 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-03-04 15:39:00.680601360 +0000 2024-03-04T15:39:07.660 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-03-04 15:26:35.933710095 +0000 2024-03-04T15:39:07.660 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-03-04 15:26:35.933710095 +0000 2024-03-04T15:39:07.660 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: 2024-03-04 15:26:35.921710354 +0000 2024-03-04T15:39:07.660 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-04T15:39:07.725 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-03-04T15:39:07.725 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-03-04T15:39:07.725 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000212939 s, 2.4 MB/s 2024-03-04T15:39:07.726 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-04T15:39:07.782 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-04T15:39:07.782 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:39:07.798 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-04T15:39:07.798 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/nvme1n1 2024-03-04T15:39:07.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:07 smithi080 ceph-mon[27564]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:07.854 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/nvme1n1 2024-03-04T15:39:07.854 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:39:07.854 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 764 Links: 1 Device type: 103,2 2024-03-04T15:39:07.854 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:39:07.854 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:39:07.854 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-03-04 15:39:00.540622970 +0000 2024-03-04T15:39:07.854 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-03-04 15:26:37.183896837 +0000 2024-03-04T15:39:07.854 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-03-04 15:26:37.183896837 +0000 2024-03-04T15:39:07.854 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-03-04 15:26:37.175897008 +0000 2024-03-04T15:39:07.854 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-04T15:39:07.924 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-03-04T15:39:07.925 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-03-04T15:39:07.925 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000231662 s, 2.2 MB/s 2024-03-04T15:39:07.926 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-04T15:39:07.984 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/nvme2n1 2024-03-04T15:39:08.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:07 smithi087 ceph-mon[27664]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:08.039 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/nvme2n1 2024-03-04T15:39:08.039 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:39:08.039 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-04T15:39:08.039 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:39:08.039 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:39:08.039 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-03-04 15:39:00.550622743 +0000 2024-03-04T15:39:08.040 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-03-04 15:26:37.339893507 +0000 2024-03-04T15:39:08.040 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-03-04 15:26:37.339893507 +0000 2024-03-04T15:39:08.040 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-03-04 15:26:37.329893720 +0000 2024-03-04T15:39:08.040 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-04T15:39:08.105 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-03-04T15:39:08.106 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-03-04T15:39:08.106 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000207806 s, 2.5 MB/s 2024-03-04T15:39:08.107 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-04T15:39:08.162 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/nvme3n1 2024-03-04T15:39:08.171 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:07 smithi005 ceph-mon[25858]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/nvme3n1 2024-03-04T15:39:08.217 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:39:08.218 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-04T15:39:08.218 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:39:08.218 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:39:08.218 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-03-04 15:39:00.561622494 +0000 2024-03-04T15:39:08.218 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-03-04 15:26:37.492890240 +0000 2024-03-04T15:39:08.218 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-03-04 15:26:37.492890240 +0000 2024-03-04T15:39:08.218 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-03-04 15:26:37.484890411 +0000 2024-03-04T15:39:08.218 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-04T15:39:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-03-04T15:39:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-03-04T15:39:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.00025435 s, 2.0 MB/s 2024-03-04T15:39:08.285 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-04T15:39:08.340 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/nvme4n1 2024-03-04T15:39:08.395 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/nvme4n1 2024-03-04T15:39:08.395 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:39:08.395 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-04T15:39:08.396 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:39:08.396 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:39:08.396 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-03-04 15:39:00.571622267 +0000 2024-03-04T15:39:08.396 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-03-04 15:26:37.647886931 +0000 2024-03-04T15:39:08.396 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-03-04 15:26:37.647886931 +0000 2024-03-04T15:39:08.396 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-03-04 15:26:37.635887187 +0000 2024-03-04T15:39:08.396 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-04T15:39:08.462 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-03-04T15:39:08.462 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-03-04T15:39:08.462 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000178358 s, 2.9 MB/s 2024-03-04T15:39:08.464 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-04T15:39:08.520 INFO:tasks.cephadm:Deploying osd.0 on smithi005 with /dev/nvme4n1... 2024-03-04T15:39:08.520 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-04T15:39:08.768 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:39:08.777 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:08 smithi087 systemd[1]: Started Ceph mgr.z for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:39:09.277 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:08 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:08.906+0000 7fdbc9e53200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:39:09.277 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:08 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:08.998+0000 7fdbc9e53200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:39:09.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:09 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:09 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:09 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:09 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:39:09.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:09 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:39:09.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:09 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:39:09.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:09 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:09.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:09 smithi005 ceph-mon[25858]: Deploying daemon mgr.x on smithi080 2024-03-04T15:39:09.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:09 smithi005 ceph-mon[25858]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:09.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:09 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:09 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.754 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:09 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.832 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:09 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.832 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:09 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.832 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:09 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.833 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:09 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:39:09.833 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:09 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:39:09.833 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:09 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:39:09.833 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:09 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:09.833 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:09 smithi080 ceph-mon[27564]: Deploying daemon mgr.x on smithi080 2024-03-04T15:39:09.833 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:09 smithi080 ceph-mon[27564]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:09.833 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:09 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.833 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:09 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.833 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:09 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.864 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:09.635+0000 7fdbc9e53200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:39:09.865 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:09.725+0000 7fdbc9e53200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:39:09.865 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:09.863+0000 7fdbc9e53200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:39:09.865 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.865 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.865 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.865 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:39:09.865 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:39:09.865 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:39:09.865 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:09.865 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-mon[27664]: Deploying daemon mgr.x on smithi080 2024-03-04T15:39:09.865 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-mon[27664]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:09.865 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.865 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:09.866 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:10.277 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:09 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:09.956+0000 7fdbc9e53200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:39:10.277 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:10 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:10.037+0000 7fdbc9e53200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:39:10.585 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:10 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:10.300+0000 7fdbc9e53200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:39:10.585 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:10 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:10.407+0000 7fdbc9e53200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:39:10.585 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:10 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:10.491+0000 7fdbc9e53200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:39:10.585 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:10 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:10.583+0000 7fdbc9e53200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:39:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping: /dev/nvme4n1 2024-03-04T15:39:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-04T15:39:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-04T15:39:10.593 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10+0 records in 2024-03-04T15:39:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2024-03-04T15:39:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00965338 s, 1.1 GB/s 2024-03-04T15:39:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping successful for: 2024-03-04T15:39:10.804 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:10 smithi080 systemd[1]: Starting Ceph mgr.x for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:39:10.855 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:10 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:10.754+0000 7fdbc9e53200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:39:11.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:10 smithi080 ceph-mon[27564]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:11.096 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:10 smithi080 podman[29239]: 2024-03-04 15:39:10.705569933 +0000 UTC m=+0.017705287 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:39:11.096 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:10 smithi080 podman[29239]: 2024-03-04 15:39:10.808720461 +0000 UTC m=+0.120855814 container create b03f3786212171e32d9c4acc8b0048f396b6fe0a274654d0edc9ee570cd528f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T15:39:11.109 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:10 smithi005 ceph-mon[25858]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:11.135 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi005:/dev/nvme4n1 2024-03-04T15:39:11.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:10 smithi087 ceph-mon[27664]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:11.397 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:39:11.474 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:11 smithi080 podman[29239]: 2024-03-04 15:39:11.102711145 +0000 UTC m=+0.414846510 container init b03f3786212171e32d9c4acc8b0048f396b6fe0a274654d0edc9ee570cd528f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-04T15:39:11.475 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:11 smithi080 podman[29239]: 2024-03-04 15:39:11.108042635 +0000 UTC m=+0.420177989 container start b03f3786212171e32d9c4acc8b0048f396b6fe0a274654d0edc9ee570cd528f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-04T15:39:11.475 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:11 smithi080 ceph-mgr[29277]: -- 172.21.15.80:0/2588532218 <== mon.2 v2:172.21.15.80:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559e2ba845a0 con 0x559e2ba58c00 2024-03-04T15:39:11.475 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:11 smithi080 bash[29239]: b03f3786212171e32d9c4acc8b0048f396b6fe0a274654d0edc9ee570cd528f7 2024-03-04T15:39:11.475 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:11 smithi080 systemd[1]: Started Ceph mgr.x for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:39:11.771 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:11.673+0000 7f0a74a11200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:39:11.771 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:11.769+0000 7f0a74a11200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:39:12.027 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:11 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:11.718+0000 7fdbc9e53200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:39:12.027 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:11 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:11.848+0000 7fdbc9e53200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:39:12.329 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:12.186+0000 7fdbc9e53200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:39:12.329 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:12.276+0000 7fdbc9e53200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:39:12.330 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.330 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.330 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.330 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.330 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:12.330 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:12.330 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:12.330 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.330 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-mon[27664]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:39:12.330 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-mon[27664]: Reconfiguring mgr.y (unknown last config time)... 2024-03-04T15:39:12.330 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:39:12.330 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:39:12.330 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:12.330 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-mon[27664]: Reconfiguring daemon mgr.y on smithi005 2024-03-04T15:39:12.331 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.554 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:12.460+0000 7f0a74a11200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:39:12.555 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:12.552+0000 7f0a74a11200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:39:12.555 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.555 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.555 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.555 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.555 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:12.555 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:12.555 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:12.555 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.555 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-mon[27564]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:39:12.555 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-mon[27564]: Reconfiguring mgr.y (unknown last config time)... 2024-03-04T15:39:12.556 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:39:12.556 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:39:12.556 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:12.556 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-mon[27564]: Reconfiguring daemon mgr.y on smithi005 2024-03-04T15:39:12.556 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:12 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:12 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:12 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:12 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:12 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:12.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:12 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:12.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:12 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:12.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:12 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:12 smithi005 ceph-mon[25858]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:39:12.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:12 smithi005 ceph-mon[25858]: Reconfiguring mgr.y (unknown last config time)... 2024-03-04T15:39:12.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:12 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:39:12.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:12 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:39:12.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:12 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:12.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:12 smithi005 ceph-mon[25858]: Reconfiguring daemon mgr.y on smithi005 2024-03-04T15:39:12.576 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:12 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:12.668 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:12.485+0000 7fdbc9e53200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:39:12.668 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:12.666+0000 7fdbc9e53200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:39:12.846 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:12.701+0000 7f0a74a11200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:39:12.846 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:12.797+0000 7f0a74a11200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:39:13.152 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:12 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:12.879+0000 7f0a74a11200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:39:13.245 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:12 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:12.971+0000 7fdbc9e53200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:39:13.245 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:13 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:13.243+0000 7fdbc9e53200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:39:13.470 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:13 smithi080 ceph-mon[27564]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:13.470 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:13 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:13.169+0000 7f0a74a11200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:39:13.470 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:13 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:13.284+0000 7f0a74a11200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:39:13.470 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:13 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:13.371+0000 7f0a74a11200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:39:13.470 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:13 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:13.468+0000 7f0a74a11200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:39:13.514 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:13 smithi087 ceph-mon[27664]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:13.632 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:13 smithi005 ceph-mon[25858]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:13.767 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:13 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:13.511+0000 7fdbc9e53200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:39:13.846 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:13 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:13.665+0000 7f0a74a11200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:39:14.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:14 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:39:14.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:14 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:39:14.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:14 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:14.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:14 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:14.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:14 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:14.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:14 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:14.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:14 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:14.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:14 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:14.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:14 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:14.591 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:39:14.591 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:39:14.591 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:14.591 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:14.591 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:14.591 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:14.592 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:14.592 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:14.592 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:14.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:14 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:39:14.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:14 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:39:14.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:14 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:14.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:14 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:14.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:14 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:14.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:14 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:14.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:14 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:14.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:14 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:14.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:14 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:14.670 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:14 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:14.333+0000 7fdbc9e53200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:39:14.885 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:14 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:14.687+0000 7f0a74a11200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:39:14.885 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:14 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:14.821+0000 7f0a74a11200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:39:15.012 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:14 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:14.910+0000 7fdbc9e53200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:39:15.012 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:15.010+0000 7fdbc9e53200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:39:15.277 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:39:15.223+0000 7fdbc9e53200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:39:15.480 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:15.173+0000 7f0a74a11200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:39:15.481 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:15.283+0000 7f0a74a11200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:39:15.481 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:15.478+0000 7f0a74a11200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:39:15.730 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:39:15.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:39:15.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:15.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: Standby manager daemon z started 2024-03-04T15:39:15.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.87:0/471663773' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T15:39:15.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.87:0/471663773' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:39:15.731 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.87:0/471663773' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T15:39:15.732 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.87:0/471663773' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:39:15.732 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.732 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.732 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.732 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:15.732 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:15.732 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:15.732 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:15 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.846 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:15.660+0000 7f0a74a11200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:39:15.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:39:15.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:39:15.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:15.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: Standby manager daemon z started 2024-03-04T15:39:15.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.87:0/471663773' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T15:39:15.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.87:0/471663773' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:39:15.848 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.87:0/471663773' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T15:39:15.848 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.87:0/471663773' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:39:15.848 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.848 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.848 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:15.848 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:15.848 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:15.848 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:15.848 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:16.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:39:16.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:39:16.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:16.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:16.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:16.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:16.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:16.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:16.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: Standby manager daemon z started 2024-03-04T15:39:16.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.87:0/471663773' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T15:39:16.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.87:0/471663773' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:39:16.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.87:0/471663773' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T15:39:16.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.87:0/471663773' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:39:16.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:16.029 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:16.029 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:16.029 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:16.029 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:16.029 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:16.029 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:15 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:16.199 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:15 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:15.939+0000 7f0a74a11200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:39:16.199 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:16 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:16.197+0000 7f0a74a11200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:39:16.596 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:16 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:16.428+0000 7f0a74a11200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:39:17.346 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:17 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:17.214+0000 7f0a74a11200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:39:17.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:17 smithi080 ceph-mon[27564]: mgrmap e14: y(active, since 79s), standbys: z 2024-03-04T15:39:17.347 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:17 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T15:39:17.347 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:17 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/584338554' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8ffc2cd-fb54-4a4f-b317-f543ca3cbbfc"}]: dispatch 2024-03-04T15:39:17.347 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:17 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/584338554' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8ffc2cd-fb54-4a4f-b317-f543ca3cbbfc"}]': finished 2024-03-04T15:39:17.347 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:17 smithi080 ceph-mon[27564]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T15:39:17.347 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:17 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:17.347 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:17 smithi080 ceph-mon[27564]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:17 smithi005 ceph-mon[25858]: mgrmap e14: y(active, since 79s), standbys: z 2024-03-04T15:39:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:17 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T15:39:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:17 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/584338554' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8ffc2cd-fb54-4a4f-b317-f543ca3cbbfc"}]: dispatch 2024-03-04T15:39:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:17 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/584338554' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8ffc2cd-fb54-4a4f-b317-f543ca3cbbfc"}]': finished 2024-03-04T15:39:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:17 smithi005 ceph-mon[25858]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T15:39:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:17 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:17 smithi005 ceph-mon[25858]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:17.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:17 smithi087 ceph-mon[27664]: mgrmap e14: y(active, since 79s), standbys: z 2024-03-04T15:39:17.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:17 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T15:39:17.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:17 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/584338554' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8ffc2cd-fb54-4a4f-b317-f543ca3cbbfc"}]: dispatch 2024-03-04T15:39:17.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:17 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/584338554' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8ffc2cd-fb54-4a4f-b317-f543ca3cbbfc"}]': finished 2024-03-04T15:39:17.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:17 smithi087 ceph-mon[27664]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T15:39:17.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:17 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:17.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:17 smithi087 ceph-mon[27664]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:18.084 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:17 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:17.775+0000 7f0a74a11200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:39:18.084 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:17 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:17.872+0000 7f0a74a11200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:39:18.084 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:39:18 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:39:18.082+0000 7f0a74a11200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:39:18.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:18 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/1659306048' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:39:18.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:18 smithi080 ceph-mon[27564]: Standby manager daemon x started 2024-03-04T15:39:18.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:18 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.80:0/33585524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:39:18.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:18 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.80:0/33585524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:39:18.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:18 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.80:0/33585524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:39:18.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:18 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.80:0/33585524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:39:18.873 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:18 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/1659306048' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:39:18.873 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:18 smithi087 ceph-mon[27664]: Standby manager daemon x started 2024-03-04T15:39:18.873 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:18 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.80:0/33585524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:39:18.874 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:18 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.80:0/33585524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:39:18.874 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:18 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.80:0/33585524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:39:18.874 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:18 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.80:0/33585524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:39:18.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:18 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/1659306048' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:39:18.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:18 smithi005 ceph-mon[25858]: Standby manager daemon x started 2024-03-04T15:39:18.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:18 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.80:0/33585524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:39:18.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:18 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.80:0/33585524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:39:18.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:18 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.80:0/33585524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:39:18.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:18 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.80:0/33585524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:39:19.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:19 smithi080 ceph-mon[27564]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-04T15:39:19.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:19 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:39:19.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:19 smithi080 ceph-mon[27564]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:19.930 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:19 smithi005 ceph-mon[25858]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-04T15:39:19.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:19 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:39:19.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:19 smithi005 ceph-mon[25858]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:20.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:19 smithi087 ceph-mon[27664]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-04T15:39:20.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:19 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:39:20.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:19 smithi087 ceph-mon[27664]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:21.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:20 smithi005 ceph-mon[25858]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:21.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:20 smithi087 ceph-mon[27664]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:21.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:20 smithi080 ceph-mon[27564]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:23.067 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:22 smithi005 ceph-mon[25858]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:23.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:22 smithi087 ceph-mon[27664]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:23.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:22 smithi080 ceph-mon[27564]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:24.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T15:39:24.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:24.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:23 smithi005 ceph-mon[25858]: Deploying daemon osd.0 on smithi005 2024-03-04T15:39:24.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:23 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T15:39:24.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:23 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:24.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:23 smithi087 ceph-mon[27664]: Deploying daemon osd.0 on smithi005 2024-03-04T15:39:24.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:23 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T15:39:24.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:23 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:24.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:23 smithi080 ceph-mon[27564]: Deploying daemon osd.0 on smithi005 2024-03-04T15:39:25.160 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:24 smithi005 ceph-mon[25858]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:25.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:24 smithi087 ceph-mon[27664]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:25.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:24 smithi080 ceph-mon[27564]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:26.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:26 smithi005 ceph-mon[25858]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:27.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:26 smithi087 ceph-mon[27664]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:27.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:26 smithi080 ceph-mon[27564]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:28.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:28 smithi005 ceph-mon[25858]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:29.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:28 smithi087 ceph-mon[27664]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:29.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:28 smithi080 ceph-mon[27564]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:30.331 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:30 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:30.331 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:30 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:30.331 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:30 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:30.331 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:30 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:30.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:30 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:30.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:30 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:30.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:30 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:30.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:30 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:30.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:30 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:30.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:30 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:30.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:30 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:30.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:30 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:31.212 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:31 smithi005 ceph-mon[25858]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:31.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:31 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:31 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:31.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:31 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:31.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:31 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:31.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:31 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:31 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:31 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:31 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:31 smithi087 ceph-mon[27664]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:31.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:31 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:31 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:31.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:31 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:31.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:31 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:31.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:31 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:31 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:31 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:31 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:31.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:31 smithi080 ceph-mon[27564]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:33.095 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:32 smithi080 ceph-mon[27564]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:33.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:32 smithi005 ceph-mon[25858]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:33.201 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:32 smithi087 ceph-mon[27664]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:33.934 INFO:teuthology.orchestra.run.smithi005.stdout:Created osd(s) 0 on host 'smithi005' 2024-03-04T15:39:34.014 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:33 smithi087 ceph-mon[27664]: from='osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:39:34.092 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:33 smithi080 ceph-mon[27564]: from='osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:39:34.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:33 smithi005 ceph-mon[25858]: from='osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:39:34.611 DEBUG:teuthology.orchestra.run.smithi005:osd.0> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@osd.0.service 2024-03-04T15:39:34.614 INFO:tasks.cephadm:Deploying osd.1 on smithi080 with /dev/nvme4n1... 2024-03-04T15:39:34.614 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-04T15:39:34.971 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T15:39:34.971 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T15:39:34.971 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-04T15:39:34.971 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:34.971 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.971 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.971 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.972 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.972 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.972 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:34.972 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:39:34.972 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: Adjusting osd_memory_target on smithi005 to 16069M 2024-03-04T15:39:34.972 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.972 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:34.972 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:34.972 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.972 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.972 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.972 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.972 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:34 smithi080 ceph-mon[27564]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:34.990 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T15:39:34.990 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T15:39:34.990 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-04T15:39:34.990 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:34.990 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.990 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.990 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.990 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.990 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.990 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:34.990 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:39:34.991 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: Adjusting osd_memory_target on smithi005 to 16069M 2024-03-04T15:39:34.991 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.991 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:34.991 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:34.991 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.991 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.991 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.991 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:34.991 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:34 smithi087 ceph-mon[27664]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:35.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T15:39:35.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T15:39:35.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-04T15:39:35.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:35.149 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:35.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:35.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:35.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:35.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:35.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:35.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:39:35.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: Adjusting osd_memory_target on smithi005 to 16069M 2024-03-04T15:39:35.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:35.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:35.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:35.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:35.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:35.150 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:35.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:35.151 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:34 smithi005 ceph-mon[25858]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:35.302 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T15:39:35.431 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:39:35 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-0[38313]: 2024-03-04T15:39:35.146+0000 7f291201a700 -1 osd.0 0 waiting for initial osdmap 2024-03-04T15:39:35.431 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:39:35 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-0[38313]: 2024-03-04T15:39:35.148+0000 7f2909e05700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:39:36.034 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:35 smithi080 ceph-mon[27564]: from='osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-03-04T15:39:36.034 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:35 smithi080 ceph-mon[27564]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T15:39:36.034 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:36.034 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:36.034 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:35 smithi080 ceph-mon[27564]: from='osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915]' entity='osd.0' 2024-03-04T15:39:36.034 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:36.178 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:35 smithi087 ceph-mon[27664]: from='osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-03-04T15:39:36.179 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:35 smithi087 ceph-mon[27664]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T15:39:36.179 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:36.179 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:36.179 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:35 smithi087 ceph-mon[27664]: from='osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915]' entity='osd.0' 2024-03-04T15:39:36.179 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:36.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:35 smithi005 ceph-mon[25858]: from='osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-03-04T15:39:36.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:35 smithi005 ceph-mon[25858]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T15:39:36.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:36.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:36.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:35 smithi005 ceph-mon[25858]: from='osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915]' entity='osd.0' 2024-03-04T15:39:36.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:37.236 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping: /dev/nvme4n1 2024-03-04T15:39:37.236 INFO:teuthology.orchestra.run.smithi080.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-04T15:39:37.236 INFO:teuthology.orchestra.run.smithi080.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-04T15:39:37.237 INFO:teuthology.orchestra.run.smithi080.stderr: stderr: 10+0 records in 2024-03-04T15:39:37.237 INFO:teuthology.orchestra.run.smithi080.stderr:10+0 records out 2024-03-04T15:39:37.237 INFO:teuthology.orchestra.run.smithi080.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126022 s, 832 MB/s 2024-03-04T15:39:37.237 INFO:teuthology.orchestra.run.smithi080.stderr:--> Zapping successful for: 2024-03-04T15:39:37.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:37 smithi005 ceph-mon[25858]: purged_snaps scrub starts 2024-03-04T15:39:37.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:37 smithi005 ceph-mon[25858]: purged_snaps scrub ok 2024-03-04T15:39:37.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:37 smithi005 ceph-mon[25858]: osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915] boot 2024-03-04T15:39:37.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:37 smithi005 ceph-mon[25858]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T15:39:37.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:37 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:37.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:37 smithi005 ceph-mon[25858]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:37.482 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:37 smithi080 ceph-mon[27564]: purged_snaps scrub starts 2024-03-04T15:39:37.482 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:37 smithi080 ceph-mon[27564]: purged_snaps scrub ok 2024-03-04T15:39:37.482 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:37 smithi080 ceph-mon[27564]: osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915] boot 2024-03-04T15:39:37.482 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:37 smithi080 ceph-mon[27564]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T15:39:37.482 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:37 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:37.482 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:37 smithi080 ceph-mon[27564]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:37.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:37 smithi087 ceph-mon[27664]: purged_snaps scrub starts 2024-03-04T15:39:37.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:37 smithi087 ceph-mon[27664]: purged_snaps scrub ok 2024-03-04T15:39:37.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:37 smithi087 ceph-mon[27664]: osd.0 [v2:172.21.15.5:6802/3224525915,v1:172.21.15.5:6803/3224525915] boot 2024-03-04T15:39:37.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:37 smithi087 ceph-mon[27664]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T15:39:37.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:37 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:39:37.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:37 smithi087 ceph-mon[27664]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:39:37.817 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi080:/dev/nvme4n1 2024-03-04T15:39:38.049 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T15:39:38.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:38 smithi005 ceph-mon[25858]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T15:39:38.468 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:38 smithi080 ceph-mon[27564]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T15:39:38.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:38 smithi087 ceph-mon[27664]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T15:39:39.424 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:39 smithi080 ceph-mon[27564]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:39.430 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:39 smithi005 ceph-mon[25858]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:39.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:39 smithi087 ceph-mon[27664]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:40.427 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:40 smithi080 ceph-mon[27564]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:39:40.427 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:40 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:39:40.427 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:40 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:39:40.427 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:40 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:40.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:40 smithi005 ceph-mon[25858]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:39:40.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:40 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:39:40.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:40 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:39:40.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:40 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:40.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:40 smithi087 ceph-mon[27664]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi080:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:39:40.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:40 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:39:40.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:40 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:39:40.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:40 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:41.271 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:41 smithi080 ceph-mon[27564]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:41.430 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:41 smithi005 ceph-mon[25858]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:41.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:41 smithi087 ceph-mon[27664]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:42.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:42 smithi087 ceph-mon[27664]: from='client.? 172.21.15.80:0/4198443305' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f5f296e-1fb4-410f-90a2-809ffcc01724"}]: dispatch 2024-03-04T15:39:42.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:42 smithi087 ceph-mon[27664]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f5f296e-1fb4-410f-90a2-809ffcc01724"}]: dispatch 2024-03-04T15:39:42.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:42 smithi087 ceph-mon[27664]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f5f296e-1fb4-410f-90a2-809ffcc01724"}]': finished 2024-03-04T15:39:42.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:42 smithi087 ceph-mon[27664]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T15:39:42.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:42 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:39:42.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:42 smithi080 ceph-mon[27564]: from='client.? 172.21.15.80:0/4198443305' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f5f296e-1fb4-410f-90a2-809ffcc01724"}]: dispatch 2024-03-04T15:39:42.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:42 smithi080 ceph-mon[27564]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f5f296e-1fb4-410f-90a2-809ffcc01724"}]: dispatch 2024-03-04T15:39:42.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:42 smithi080 ceph-mon[27564]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f5f296e-1fb4-410f-90a2-809ffcc01724"}]': finished 2024-03-04T15:39:42.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:42 smithi080 ceph-mon[27564]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T15:39:42.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:42 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:39:42.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:42 smithi005 ceph-mon[25858]: from='client.? 172.21.15.80:0/4198443305' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f5f296e-1fb4-410f-90a2-809ffcc01724"}]: dispatch 2024-03-04T15:39:42.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:42 smithi005 ceph-mon[25858]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f5f296e-1fb4-410f-90a2-809ffcc01724"}]: dispatch 2024-03-04T15:39:42.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:42 smithi005 ceph-mon[25858]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f5f296e-1fb4-410f-90a2-809ffcc01724"}]': finished 2024-03-04T15:39:42.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:42 smithi005 ceph-mon[25858]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T15:39:42.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:42 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:39:43.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:43 smithi087 ceph-mon[27664]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:43.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:43 smithi080 ceph-mon[27564]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:43.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:43 smithi005 ceph-mon[25858]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:44.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:44 smithi087 ceph-mon[27664]: from='client.? 172.21.15.80:0/2996658735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:39:44.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:44 smithi080 ceph-mon[27564]: from='client.? 172.21.15.80:0/2996658735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:39:44.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:44 smithi005 ceph-mon[25858]: from='client.? 172.21.15.80:0/2996658735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:39:45.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:45 smithi087 ceph-mon[27664]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:45.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:45 smithi080 ceph-mon[27564]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:45.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:45 smithi005 ceph-mon[25858]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:47.146 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:46 smithi080 ceph-mon[27564]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:47.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:46 smithi005 ceph-mon[25858]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:47.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:46 smithi087 ceph-mon[27664]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:49.009 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:48 smithi080 ceph-mon[27564]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:49.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:48 smithi005 ceph-mon[25858]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:49.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:48 smithi087 ceph-mon[27664]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:50.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:49 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:39:50.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:49 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:50.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:49 smithi005 ceph-mon[25858]: Deploying daemon osd.1 on smithi080 2024-03-04T15:39:50.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:49 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:50.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:49 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:50.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:49 smithi005 ceph-mon[25858]: Detected new or changed devices on smithi005 2024-03-04T15:39:50.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:49 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:50.220 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:49 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:39:50.220 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:49 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:50.220 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:49 smithi080 ceph-mon[27564]: Deploying daemon osd.1 on smithi080 2024-03-04T15:39:50.220 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:49 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:50.220 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:49 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:50.221 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:49 smithi080 ceph-mon[27564]: Detected new or changed devices on smithi005 2024-03-04T15:39:50.221 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:49 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:50.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:49 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:39:50.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:49 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:50.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:49 smithi087 ceph-mon[27664]: Deploying daemon osd.1 on smithi080 2024-03-04T15:39:50.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:49 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:50.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:49 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:50.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:49 smithi087 ceph-mon[27664]: Detected new or changed devices on smithi005 2024-03-04T15:39:50.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:49 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:50.930 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:50 smithi005 ceph-mon[25858]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:51.095 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:50 smithi080 ceph-mon[27564]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:51.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:50 smithi087 ceph-mon[27664]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:53.024 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:52 smithi080 ceph-mon[27564]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:53.122 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:52 smithi087 ceph-mon[27664]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:53.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:52 smithi005 ceph-mon[25858]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:55.254 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.254 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.255 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.255 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.255 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.255 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.255 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:55.255 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.255 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.255 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:55.255 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.255 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:55.255 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:55.255 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.255 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:39:55.255 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:55.416 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.416 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:55.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:55.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:55.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:55.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:39:55.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:55.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:55.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:39:55.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:55.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:39:55.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:55.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:39:55.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:39:56.290 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:56 smithi080 ceph-mon[27564]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:56.290 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:56 smithi080 ceph-mon[27564]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:56.290 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:56 smithi080 ceph-mon[27564]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:56.290 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:56 smithi080 ceph-mon[27564]: Reconfiguring osd.1 (unknown last config time)... 2024-03-04T15:39:56.290 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:56 smithi080 ceph-mon[27564]: Reconfiguring daemon osd.1 on smithi080 2024-03-04T15:39:56.290 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:56 smithi080 ceph-mon[27564]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-04T15:39:56.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:56 smithi005 ceph-mon[25858]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:56.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:56 smithi005 ceph-mon[25858]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:56.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:56 smithi005 ceph-mon[25858]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:56.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:56 smithi005 ceph-mon[25858]: Reconfiguring osd.1 (unknown last config time)... 2024-03-04T15:39:56.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:56 smithi005 ceph-mon[25858]: Reconfiguring daemon osd.1 on smithi080 2024-03-04T15:39:56.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:56 smithi005 ceph-mon[25858]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-04T15:39:56.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:56 smithi087 ceph-mon[27664]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:56.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:56 smithi087 ceph-mon[27664]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:56.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:56 smithi087 ceph-mon[27664]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:56.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:56 smithi087 ceph-mon[27664]: Reconfiguring osd.1 (unknown last config time)... 2024-03-04T15:39:56.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:56 smithi087 ceph-mon[27664]: Reconfiguring daemon osd.1 on smithi080 2024-03-04T15:39:56.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:56 smithi087 ceph-mon[27664]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-04T15:39:57.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:57.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:57 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:57.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:57.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:57 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:57.592 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:57 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:57.592 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:57 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:39:58.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:39:58 smithi080 ceph-mon[27564]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:58.430 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:39:58 smithi005 ceph-mon[25858]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:58.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:39:58 smithi087 ceph-mon[27664]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:39:59.954 INFO:teuthology.orchestra.run.smithi080.stdout:Created osd(s) 1 on host 'smithi080' 2024-03-04T15:40:00.197 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:00 smithi080 ceph-mon[27564]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:40:00.198 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:00 smithi080 ceph-mon[27564]: from='osd.1 [v2:172.21.15.80:6800/1635610381,v1:172.21.15.80:6801/1635610381]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:40:00.198 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:00 smithi080 ceph-mon[27564]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:40:00.198 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:00 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:00.198 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:00 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:00.198 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:00 smithi080 ceph-mon[27564]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-04T15:40:00.198 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:00 smithi080 ceph-mon[27564]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-04T15:40:00.198 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:00 smithi080 ceph-mon[27564]: daemon osd.1 on smithi080 is in unknown state 2024-03-04T15:40:00.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:00 smithi005 ceph-mon[25858]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:40:00.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:00 smithi005 ceph-mon[25858]: from='osd.1 [v2:172.21.15.80:6800/1635610381,v1:172.21.15.80:6801/1635610381]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:40:00.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:00 smithi005 ceph-mon[25858]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:40:00.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:00 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:00.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:00 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:00.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:00 smithi005 ceph-mon[25858]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-04T15:40:00.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:00 smithi005 ceph-mon[25858]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-04T15:40:00.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:00 smithi005 ceph-mon[25858]: daemon osd.1 on smithi080 is in unknown state 2024-03-04T15:40:00.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:00 smithi087 ceph-mon[27664]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:40:00.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:00 smithi087 ceph-mon[27664]: from='osd.1 [v2:172.21.15.80:6800/1635610381,v1:172.21.15.80:6801/1635610381]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:40:00.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:00 smithi087 ceph-mon[27664]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:40:00.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:00 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:00.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:00 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:00.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:00 smithi087 ceph-mon[27664]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-04T15:40:00.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:00 smithi087 ceph-mon[27664]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-04T15:40:00.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:00 smithi087 ceph-mon[27664]: daemon osd.1 on smithi080 is in unknown state 2024-03-04T15:40:00.807 DEBUG:teuthology.orchestra.run.smithi080:osd.1> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@osd.1.service 2024-03-04T15:40:00.809 INFO:tasks.cephadm:Deploying osd.2 on smithi087 with /dev/nvme4n1... 2024-03-04T15:40:00.809 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-04T15:40:01.059 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:40:01.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:01 smithi005 ceph-mon[25858]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T15:40:01.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:01 smithi005 ceph-mon[25858]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T15:40:01.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:01 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:40:01.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:01 smithi005 ceph-mon[25858]: from='osd.1 [v2:172.21.15.80:6800/1635610381,v1:172.21.15.80:6801/1635610381]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-04T15:40:01.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:01 smithi005 ceph-mon[25858]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-04T15:40:01.436 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:01 smithi087 ceph-mon[27664]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T15:40:01.436 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:01 smithi087 ceph-mon[27664]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T15:40:01.436 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:01 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:40:01.436 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:01 smithi087 ceph-mon[27664]: from='osd.1 [v2:172.21.15.80:6800/1635610381,v1:172.21.15.80:6801/1635610381]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-04T15:40:01.437 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:01 smithi087 ceph-mon[27664]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-04T15:40:01.509 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:01 smithi080 ceph-mon[27564]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T15:40:01.509 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:01 smithi080 ceph-mon[27564]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T15:40:01.509 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:01 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:40:01.509 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:01 smithi080 ceph-mon[27564]: from='osd.1 [v2:172.21.15.80:6800/1635610381,v1:172.21.15.80:6801/1635610381]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-04T15:40:01.509 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:01 smithi080 ceph-mon[27564]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]: dispatch 2024-03-04T15:40:01.846 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:40:01 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-1[33185]: 2024-03-04T15:40:01.623+0000 7ff44b746700 -1 osd.1 0 waiting for initial osdmap 2024-03-04T15:40:01.846 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:40:01 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-1[33185]: 2024-03-04T15:40:01.625+0000 7ff443531700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:40:02.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:40:02.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-04T15:40:02.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T15:40:02.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:40:02.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:40:02.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:02.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:02.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: from='osd.1 ' entity='osd.1' 2024-03-04T15:40:02.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:02.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:40:02.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: Adjusting osd_memory_target on smithi080 to 16064M 2024-03-04T15:40:02.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:02.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:02.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:02.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:40:02.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:02 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:02.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:40:02.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-04T15:40:02.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T15:40:02.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:40:02.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:40:02.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:02.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:02.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: from='osd.1 ' entity='osd.1' 2024-03-04T15:40:02.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:02.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:40:02.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: Adjusting osd_memory_target on smithi080 to 16064M 2024-03-04T15:40:02.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:02.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:02.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:02.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:40:02.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:02 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:02.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:40:02.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi080", "root=default"]}]': finished 2024-03-04T15:40:02.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T15:40:02.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:40:02.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:40:02.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:02.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:02.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: from='osd.1 ' entity='osd.1' 2024-03-04T15:40:02.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:02.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:40:02.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: Adjusting osd_memory_target on smithi080 to 16064M 2024-03-04T15:40:02.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:02.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:02.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:02.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:40:02.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:02 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:02.988 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping: /dev/nvme4n1 2024-03-04T15:40:02.988 INFO:teuthology.orchestra.run.smithi087.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-04T15:40:02.988 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-04T15:40:02.988 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10+0 records in 2024-03-04T15:40:02.988 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records out 2024-03-04T15:40:02.988 INFO:teuthology.orchestra.run.smithi087.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126024 s, 832 MB/s 2024-03-04T15:40:02.988 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 2024-03-04T15:40:02.989 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping successful for: 2024-03-04T15:40:03.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:03 smithi087 ceph-mon[27664]: purged_snaps scrub starts 2024-03-04T15:40:03.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:03 smithi087 ceph-mon[27664]: purged_snaps scrub ok 2024-03-04T15:40:03.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:03 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:40:03.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:03 smithi087 ceph-mon[27664]: osd.1 [v2:172.21.15.80:6800/1635610381,v1:172.21.15.80:6801/1635610381] boot 2024-03-04T15:40:03.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:03 smithi087 ceph-mon[27664]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T15:40:03.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:03 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:40:03.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:03 smithi087 ceph-mon[27664]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-04T15:40:03.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:03 smithi087 ceph-mon[27664]: Cluster is now healthy 2024-03-04T15:40:03.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:03 smithi005 ceph-mon[25858]: purged_snaps scrub starts 2024-03-04T15:40:03.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:03 smithi005 ceph-mon[25858]: purged_snaps scrub ok 2024-03-04T15:40:03.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:03 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:40:03.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:03 smithi005 ceph-mon[25858]: osd.1 [v2:172.21.15.80:6800/1635610381,v1:172.21.15.80:6801/1635610381] boot 2024-03-04T15:40:03.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:03 smithi005 ceph-mon[25858]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T15:40:03.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:03 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:40:03.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:03 smithi005 ceph-mon[25858]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-04T15:40:03.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:03 smithi005 ceph-mon[25858]: Cluster is now healthy 2024-03-04T15:40:03.480 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi087:/dev/nvme4n1 2024-03-04T15:40:03.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:03 smithi080 ceph-mon[27564]: purged_snaps scrub starts 2024-03-04T15:40:03.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:03 smithi080 ceph-mon[27564]: purged_snaps scrub ok 2024-03-04T15:40:03.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:03 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:40:03.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:03 smithi080 ceph-mon[27564]: osd.1 [v2:172.21.15.80:6800/1635610381,v1:172.21.15.80:6801/1635610381] boot 2024-03-04T15:40:03.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:03 smithi080 ceph-mon[27564]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T15:40:03.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:03 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:40:03.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:03 smithi080 ceph-mon[27564]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-04T15:40:03.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:03 smithi080 ceph-mon[27564]: Cluster is now healthy 2024-03-04T15:40:03.715 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:40:04.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:04 smithi005 ceph-mon[25858]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:04.466 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:04 smithi087 ceph-mon[27664]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:04.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:04 smithi080 ceph-mon[27564]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:05.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:05 smithi087 ceph-mon[27664]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T15:40:05.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:05 smithi005 ceph-mon[25858]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T15:40:05.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:05 smithi080 ceph-mon[27564]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T15:40:06.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:06 smithi087 ceph-mon[27664]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:06.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:40:06.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:40:06.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:06.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:06 smithi087 ceph-mon[27664]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:06.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:06 smithi005 ceph-mon[25858]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:06.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:40:06.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:40:06.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:06.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:06 smithi005 ceph-mon[25858]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:06.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:06 smithi080 ceph-mon[27564]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:06.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:40:06.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:40:06.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:06.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:06 smithi080 ceph-mon[27564]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:08.988 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:08 smithi005 ceph-mon[25858]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:08.988 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:08 smithi005 ceph-mon[25858]: from='client.? 172.21.15.87:0/981557792' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c4fe192-e1b9-4d9f-a93d-1b760ad76c6f"}]: dispatch 2024-03-04T15:40:08.988 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:08 smithi005 ceph-mon[25858]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c4fe192-e1b9-4d9f-a93d-1b760ad76c6f"}]: dispatch 2024-03-04T15:40:08.988 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:08 smithi005 ceph-mon[25858]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c4fe192-e1b9-4d9f-a93d-1b760ad76c6f"}]': finished 2024-03-04T15:40:08.988 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:08 smithi005 ceph-mon[25858]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T15:40:08.988 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:08 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:40:09.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:08 smithi087 ceph-mon[27664]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:09.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:08 smithi087 ceph-mon[27664]: from='client.? 172.21.15.87:0/981557792' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c4fe192-e1b9-4d9f-a93d-1b760ad76c6f"}]: dispatch 2024-03-04T15:40:09.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:08 smithi087 ceph-mon[27664]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c4fe192-e1b9-4d9f-a93d-1b760ad76c6f"}]: dispatch 2024-03-04T15:40:09.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:08 smithi087 ceph-mon[27664]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c4fe192-e1b9-4d9f-a93d-1b760ad76c6f"}]': finished 2024-03-04T15:40:09.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:08 smithi087 ceph-mon[27664]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T15:40:09.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:08 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:40:09.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:08 smithi080 ceph-mon[27564]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:09.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:08 smithi080 ceph-mon[27564]: from='client.? 172.21.15.87:0/981557792' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c4fe192-e1b9-4d9f-a93d-1b760ad76c6f"}]: dispatch 2024-03-04T15:40:09.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:08 smithi080 ceph-mon[27564]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c4fe192-e1b9-4d9f-a93d-1b760ad76c6f"}]: dispatch 2024-03-04T15:40:09.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:08 smithi080 ceph-mon[27564]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c4fe192-e1b9-4d9f-a93d-1b760ad76c6f"}]': finished 2024-03-04T15:40:09.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:08 smithi080 ceph-mon[27564]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T15:40:09.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:08 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:40:09.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:09 smithi005 ceph-mon[25858]: from='client.? 172.21.15.87:0/34848461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:40:09.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:09 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:09.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:09 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:09.894 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:09 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:10.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:09 smithi087 ceph-mon[27664]: from='client.? 172.21.15.87:0/34848461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:40:10.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:09 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:10.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:09 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:10.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:09 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:10.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:09 smithi080 ceph-mon[27564]: from='client.? 172.21.15.87:0/34848461' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:40:10.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:09 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:10.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:09 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:10.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:09 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:10.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:10 smithi080 ceph-mon[27564]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:10.930 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:10 smithi005 ceph-mon[25858]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:11.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:10 smithi087 ceph-mon[27664]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:13.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:12 smithi087 ceph-mon[27664]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:13.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:12 smithi080 ceph-mon[27564]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:13.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:12 smithi005 ceph-mon[25858]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:14.914 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:14 smithi087 ceph-mon[27664]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:14.914 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:14 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:14.914 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:14 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:14.914 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:14 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:14.914 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:14 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:14.914 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:14 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:14.914 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:14 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:15.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:14 smithi080 ceph-mon[27564]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:15.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:14 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:15.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:14 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:15.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:14 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:15.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:14 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:15.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:14 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:15.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:14 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:15.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:14 smithi005 ceph-mon[25858]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:15.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:15.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:15.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:15.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:15.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:15.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:14 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:15.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:15 smithi080 ceph-mon[27564]: Detected new or changed devices on smithi080 2024-03-04T15:40:16.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:15 smithi087 ceph-mon[27664]: Detected new or changed devices on smithi080 2024-03-04T15:40:16.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:15 smithi005 ceph-mon[25858]: Detected new or changed devices on smithi080 2024-03-04T15:40:17.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:16 smithi087 ceph-mon[27664]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:17.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:16 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T15:40:17.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:16 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:17.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:16 smithi080 ceph-mon[27564]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:17.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:16 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T15:40:17.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:16 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:17.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:16 smithi005 ceph-mon[25858]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:17.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:16 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T15:40:17.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:16 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:18.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:17 smithi087 ceph-mon[27664]: Deploying daemon osd.2 on smithi087 2024-03-04T15:40:18.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:17 smithi080 ceph-mon[27564]: Deploying daemon osd.2 on smithi087 2024-03-04T15:40:18.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:17 smithi005 ceph-mon[25858]: Deploying daemon osd.2 on smithi087 2024-03-04T15:40:18.887 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:18 smithi087 ceph-mon[27664]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:19.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:18 smithi080 ceph-mon[27564]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:19.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:18 smithi005 ceph-mon[25858]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:21.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:20 smithi087 ceph-mon[27664]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:21.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:20 smithi080 ceph-mon[27564]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:21.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:20 smithi005 ceph-mon[25858]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:23.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:22 smithi087 ceph-mon[27664]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:23.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:22 smithi080 ceph-mon[27564]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:23.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:22 smithi005 ceph-mon[25858]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:24.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:24.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:24.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:24.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:24.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:24.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:23 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:24.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:23 smithi005 ceph-mon[25858]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:24.183 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:23 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:24.183 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:23 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:24.183 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:23 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:24.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:23 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:24.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:23 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:24.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:23 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:24.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:23 smithi087 ceph-mon[27664]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:24.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:23 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:24.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:23 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:24.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:23 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:24.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:23 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:24.347 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:23 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:24.347 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:23 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:24.347 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:23 smithi080 ceph-mon[27564]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:25.953 INFO:teuthology.orchestra.run.smithi087.stdout:Created osd(s) 2 on host 'smithi087' 2024-03-04T15:40:26.553 DEBUG:teuthology.orchestra.run.smithi087:osd.2> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@osd.2.service 2024-03-04T15:40:26.555 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-04T15:40:26.555 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph osd stat -f json 2024-03-04T15:40:26.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:26 smithi087 ceph-mon[27664]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:26.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:26 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:26.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:26 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:26.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:26 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:26.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:26 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:40:26.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:26 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:26.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:26 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:26.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:26 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:26.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:26 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:26.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:26 smithi087 ceph-mon[27664]: from='osd.2 [v2:172.21.15.87:6800/609742830,v1:172.21.15.87:6801/609742830]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:40:26.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:26 smithi087 ceph-mon[27664]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:40:26.812 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:40:26.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:26 smithi005 ceph-mon[25858]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:26.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:26 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:26.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:26 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:26.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:26 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:26.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:26 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:40:26.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:26 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:26.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:26 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:26.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:26 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:26.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:26 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:26.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:26 smithi005 ceph-mon[25858]: from='osd.2 [v2:172.21.15.87:6800/609742830,v1:172.21.15.87:6801/609742830]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:40:26.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:26 smithi005 ceph-mon[25858]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:40:27.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:26 smithi080 ceph-mon[27564]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:27.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:26 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:27.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:26 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:27.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:26 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:27.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:26 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:40:27.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:26 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:27.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:26 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:27.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:26 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:27.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:26 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:27.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:26 smithi080 ceph-mon[27564]: from='osd.2 [v2:172.21.15.87:6800/609742830,v1:172.21.15.87:6801/609742830]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:40:27.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:26 smithi080 ceph-mon[27564]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:40:27.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:27 smithi005 ceph-mon[25858]: Adjusting osd_memory_target on smithi087 to 16064M 2024-03-04T15:40:27.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:27 smithi005 ceph-mon[25858]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T15:40:27.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:27 smithi005 ceph-mon[25858]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T15:40:27.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:27 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:40:27.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:27 smithi005 ceph-mon[25858]: from='osd.2 [v2:172.21.15.87:6800/609742830,v1:172.21.15.87:6801/609742830]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-03-04T15:40:27.932 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:27 smithi005 ceph-mon[25858]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-03-04T15:40:28.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:27 smithi087 ceph-mon[27664]: Adjusting osd_memory_target on smithi087 to 16064M 2024-03-04T15:40:28.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:27 smithi087 ceph-mon[27664]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T15:40:28.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:27 smithi087 ceph-mon[27664]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T15:40:28.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:27 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:40:28.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:27 smithi087 ceph-mon[27664]: from='osd.2 [v2:172.21.15.87:6800/609742830,v1:172.21.15.87:6801/609742830]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-03-04T15:40:28.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:27 smithi087 ceph-mon[27664]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-03-04T15:40:28.028 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:40:27 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-2[34276]: 2024-03-04T15:40:27.964+0000 7f2a4000a700 -1 osd.2 0 waiting for initial osdmap 2024-03-04T15:40:28.028 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:40:27 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-2[34276]: 2024-03-04T15:40:27.968+0000 7f2a3bdfd700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:40:28.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:27 smithi080 ceph-mon[27564]: Adjusting osd_memory_target on smithi087 to 16064M 2024-03-04T15:40:28.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:27 smithi080 ceph-mon[27564]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T15:40:28.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:27 smithi080 ceph-mon[27564]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T15:40:28.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:27 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:40:28.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:27 smithi080 ceph-mon[27564]: from='osd.2 [v2:172.21.15.87:6800/609742830,v1:172.21.15.87:6801/609742830]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-03-04T15:40:28.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:27 smithi080 ceph-mon[27564]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-03-04T15:40:28.719 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:40:29.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:28 smithi087 ceph-mon[27664]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:29.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:28 smithi087 ceph-mon[27664]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-03-04T15:40:29.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:28 smithi087 ceph-mon[27664]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T15:40:29.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:28 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:40:29.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:28 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:40:29.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:28 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/3626547603' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T15:40:29.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:28 smithi005 ceph-mon[25858]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:29.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:28 smithi005 ceph-mon[25858]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-03-04T15:40:29.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:28 smithi005 ceph-mon[25858]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T15:40:29.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:28 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:40:29.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:28 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:40:29.092 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:28 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/3626547603' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T15:40:29.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:28 smithi080 ceph-mon[27564]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:29.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:28 smithi080 ceph-mon[27564]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-03-04T15:40:29.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:28 smithi080 ceph-mon[27564]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T15:40:29.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:28 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:40:29.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:28 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:40:29.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:28 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/3626547603' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T15:40:29.468 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":17,"num_osds":3,"num_up_osds":2,"osd_up_since":1709566802,"num_in_osds":3,"osd_in_since":1709566808,"num_remapped_pgs":0} 2024-03-04T15:40:30.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:29 smithi087 ceph-mon[27664]: purged_snaps scrub starts 2024-03-04T15:40:30.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:29 smithi087 ceph-mon[27664]: purged_snaps scrub ok 2024-03-04T15:40:30.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:29 smithi087 ceph-mon[27664]: osd.2 [v2:172.21.15.87:6800/609742830,v1:172.21.15.87:6801/609742830] boot 2024-03-04T15:40:30.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:29 smithi087 ceph-mon[27664]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T15:40:30.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:29 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:40:30.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:29 smithi087 ceph-mon[27664]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:30.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:29 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:30.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:29 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:30.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:29 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:30.340 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:29 smithi005 ceph-mon[25858]: purged_snaps scrub starts 2024-03-04T15:40:30.341 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:29 smithi005 ceph-mon[25858]: purged_snaps scrub ok 2024-03-04T15:40:30.341 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:29 smithi005 ceph-mon[25858]: osd.2 [v2:172.21.15.87:6800/609742830,v1:172.21.15.87:6801/609742830] boot 2024-03-04T15:40:30.341 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:29 smithi005 ceph-mon[25858]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T15:40:30.341 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:29 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:40:30.341 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:29 smithi005 ceph-mon[25858]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:30.341 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:29 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:30.341 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:29 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:30.341 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:29 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:30.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:29 smithi080 ceph-mon[27564]: purged_snaps scrub starts 2024-03-04T15:40:30.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:29 smithi080 ceph-mon[27564]: purged_snaps scrub ok 2024-03-04T15:40:30.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:29 smithi080 ceph-mon[27564]: osd.2 [v2:172.21.15.87:6800/609742830,v1:172.21.15.87:6801/609742830] boot 2024-03-04T15:40:30.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:29 smithi080 ceph-mon[27564]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T15:40:30.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:29 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:40:30.347 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:29 smithi080 ceph-mon[27564]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:40:30.347 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:29 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:30.347 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:29 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:30.347 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:29 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:30.470 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph osd stat -f json 2024-03-04T15:40:30.769 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:40:31.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:31 smithi005 ceph-mon[25858]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T15:40:31.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:31 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:40:31.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:31 smithi087 ceph-mon[27664]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T15:40:31.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:31 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:40:31.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:31 smithi080 ceph-mon[27564]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T15:40:31.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:31 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:40:32.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:32 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T15:40:32.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:32 smithi005 ceph-mon[25858]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T15:40:32.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:32 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:40:32.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:32 smithi005 ceph-mon[25858]: pgmap v91: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:40:32.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:32 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T15:40:32.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:32 smithi087 ceph-mon[27664]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T15:40:32.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:32 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:40:32.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:32 smithi087 ceph-mon[27664]: pgmap v91: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:40:32.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:32 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T15:40:32.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:32 smithi080 ceph-mon[27564]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T15:40:32.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:32 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:40:32.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:32 smithi080 ceph-mon[27564]: pgmap v91: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:40:32.681 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:40:32 smithi005 sudo[40621]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-04T15:40:32.681 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:40:32 smithi005 sudo[40621]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:40:32.681 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:40:32 smithi005 sudo[40621]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:40:32.681 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:40:32 smithi005 sudo[40621]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:40:33.147 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:40:33.147 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:40:32 smithi005 sudo[40644]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-04T15:40:33.147 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:40:32 smithi005 sudo[40644]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:40:33.147 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:40:32 smithi005 sudo[40644]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:40:33.147 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:40:32 smithi005 sudo[40644]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:40:33.199 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:40:33 smithi080 sudo[34795]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-04T15:40:33.199 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:40:33 smithi080 sudo[34795]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:40:33.199 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:40:33 smithi080 sudo[34795]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:40:33.199 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:40:33 smithi080 sudo[34795]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:40:33.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:33 smithi005 ceph-mon[25858]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T15:40:33.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T15:40:33.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:33 smithi005 ceph-mon[25858]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T15:40:33.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:33.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:33.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:33.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:33 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:33.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:33 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/4246737342' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T15:40:33.491 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:33 smithi087 ceph-mon[27664]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T15:40:33.491 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:33 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T15:40:33.491 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:33 smithi087 ceph-mon[27664]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T15:40:33.491 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:33 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:33.491 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:33 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:33.491 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:33 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:33.491 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:33 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:33.491 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:33 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/4246737342' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T15:40:33.596 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:40:33 smithi080 sudo[34798]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-04T15:40:33.596 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:40:33 smithi080 sudo[34798]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:40:33.596 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:40:33 smithi080 sudo[34798]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:40:33.596 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:40:33 smithi080 sudo[34798]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:40:33.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:33 smithi080 ceph-mon[27564]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T15:40:33.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:33 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T15:40:33.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:33 smithi080 ceph-mon[27564]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T15:40:33.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:33 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:33.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:33 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:33.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:33 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:33.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:33 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:33.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:33 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/4246737342' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T15:40:33.777 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:40:33 smithi087 sudo[35236]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-04T15:40:33.777 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:40:33 smithi087 sudo[35236]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:40:33.778 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:40:33 smithi087 sudo[35236]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:40:33.778 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:40:33 smithi087 sudo[35236]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:40:33.860 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":21,"num_osds":3,"num_up_osds":3,"osd_up_since":1709566828,"num_in_osds":3,"osd_in_since":1709566808,"num_remapped_pgs":0} 2024-03-04T15:40:33.860 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T15:40:34.105 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:40:34.264 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:40:33 smithi087 sudo[35299]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-04T15:40:34.264 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:40:33 smithi087 sudo[35299]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:40:34.265 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:40:33 smithi087 sudo[35299]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:40:34.265 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:40:33 smithi087 sudo[35299]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:40:34.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:34 smithi005 sudo[40769]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:40:34.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:34 smithi005 sudo[40769]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:40:34.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:34 smithi005 sudo[40769]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:40:34.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:34 smithi005 sudo[40769]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:40:34.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:34 smithi087 ceph-mon[27664]: osdmap e22: 3 total, 3 up, 3 in 2024-03-04T15:40:34.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:34.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:34 smithi087 ceph-mon[27664]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:40:34.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:34 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:34.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:34 smithi087 ceph-mon[27664]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:40:34.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:34 smithi087 ceph-mon[27664]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:40:34.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:34 smithi080 ceph-mon[27564]: osdmap e22: 3 total, 3 up, 3 in 2024-03-04T15:40:34.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:34.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:34 smithi080 ceph-mon[27564]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:40:34.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:34 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:34.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:34 smithi080 ceph-mon[27564]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:40:34.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:34 smithi080 ceph-mon[27564]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:40:34.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:34 smithi005 sudo[40810]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:40:34.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:34 smithi005 sudo[40810]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:40:34.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:34 smithi005 sudo[40810]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:40:34.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:34 smithi005 sudo[40810]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:40:34.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:34 smithi005 ceph-mon[25858]: osdmap e22: 3 total, 3 up, 3 in 2024-03-04T15:40:34.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:34.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:34 smithi005 ceph-mon[25858]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:40:34.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:34 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:34.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:34 smithi005 ceph-mon[25858]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:40:34.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:34 smithi005 ceph-mon[25858]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:40:35.011 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:34 smithi087 sudo[35408]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:40:35.011 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:34 smithi087 sudo[35408]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:40:35.011 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:34 smithi087 sudo[35408]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:40:35.011 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:34 smithi087 sudo[35408]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:40:35.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 sudo[35430]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:40:35.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 sudo[35430]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:40:35.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 sudo[35430]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:40:35.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 sudo[35430]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:40:35.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 sudo[34866]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:40:35.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 sudo[34866]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:40:35.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 sudo[34866]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:40:35.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 sudo[34866]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:40:35.743 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T15:40:35.743 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: Cluster is now healthy 2024-03-04T15:40:35.743 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:40:35.743 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:40:35.743 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:40:35.743 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-04T15:40:35.744 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.744 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:40:35.744 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:40:35.744 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:40:35.744 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:40:35.744 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:40:35.744 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.744 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.744 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.744 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:35.744 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:35.744 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:35.745 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.745 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.745 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:40:35.745 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:40:35.745 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:35 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:40:35.816 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 sudo[34889]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:40:35.816 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 sudo[34889]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:40:35.816 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 sudo[34889]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:40:35.816 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 sudo[34889]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:40:35.816 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T15:40:35.816 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: Cluster is now healthy 2024-03-04T15:40:35.816 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.817 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.818 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:40:35.818 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:40:35.818 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:35 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:40:35.898 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:40:35.898 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"fsid":"97c626fc-da3b-11ee-95c6-87774f69a715","created":"2024-03-04T15:37:02.677902+0000","modified":"2024-03-04T15:40:33.193126+0000","last_up_change":"2024-03-04T15:40:28.959524+0000","last_in_change":"2024-03-04T15:40:08.226352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:40:31.136454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e8ffc2cd-fb54-4a4f-b317-f543ca3cbbfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":3224525915},{"type":"v1","addr":"172.21.15.5:6803","nonce":3224525915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":3224525915},{"type":"v1","addr":"172.21.15.5:6805","nonce":3224525915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":3224525915},{"type":"v1","addr":"172.21.15.5:6809","nonce":3224525915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":3224525915},{"type":"v1","addr":"172.21.15.5:6807","nonce":3224525915}]},"public_addr":"172.21.15.5:6803/3224525915","cluster_addr":"172.21.15.5:6805/3224525915","heartbeat_back_addr":"172.21.15.5:6809/3224525915","heartbeat_front_addr":"172.21.15.5:6807/3224525915","state":["exists","up"]},{"osd":1,"uuid":"6f5f296e-1fb4-410f-90a2-809ffcc01724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":1635610381},{"type":"v1","addr":"172.21.15.80:6801","nonce":1635610381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":1635610381},{"type":"v1","addr":"172.21.15.80:6803","nonce":1635610381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":1635610381},{"type":"v1","addr":"172.21.15.80:6807","nonce":1635610381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":1635610381},{"type":"v1","addr":"172.21.15.80:6805","nonce":1635610381}]},"public_addr":"172.21.15.80:6801/1635610381","cluster_addr":"172.21.15.80:6803/1635610381","heartbeat_back_addr":"172.21.15.80:6807/1635610381","heartbeat_front_addr":"172.21.15.80:6805/1635610381","state":["exists","up"]},{"osd":2,"uuid":"6c4fe192-e1b9-4d9f-a93d-1b760ad76c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":609742830},{"type":"v1","addr":"172.21.15.87:6801","nonce":609742830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":609742830},{"type":"v1","addr":"172.21.15.87:6803","nonce":609742830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":609742830},{"type":"v1","addr":"172.21.15.87:6807","nonce":609742830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":609742830},{"type":"v1","addr":"172.21.15.87:6805","nonce":609742830}]},"public_addr":"172.21.15.87:6801/609742830","cluster_addr":"172.21.15.87:6803/609742830","heartbeat_back_addr":"172.21.15.87:6807/609742830","heartbeat_front_addr":"172.21.15.87:6805/609742830","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:39:34.316398+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:40:00.254436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:40:26.922481+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/1666734806":"2024-03-05T15:37:56.775816+0000","172.21.15.5:0/1558141873":"2024-03-05T15:37:56.775816+0000","172.21.15.5:0/2354875202":"2024-03-05T15:37:56.775816+0000","172.21.15.5:0/1979102031":"2024-03-05T15:37:29.460739+0000","172.21.15.5:0/3924784696":"2024-03-05T15:37:29.460739+0000","172.21.15.5:6800/3259340716":"2024-03-05T15:37:29.460739+0000","172.21.15.5:6801/4290743325":"2024-03-05T15:37:56.775816+0000","172.21.15.5:6801/3259340716":"2024-03-05T15:37:29.460739+0000","172.21.15.5:0/3674674128":"2024-03-05T15:37:29.460739+0000","172.21.15.5:6800/4290743325":"2024-03-05T15:37:56.775816+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T15:40:35.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T15:40:35.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: Cluster is now healthy 2024-03-04T15:40:35.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:40:35.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:40:35.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:40:35.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-04T15:40:35.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:40:35.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:40:35.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:40:35.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:40:35.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:40:35.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:35.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:35.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:35.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:35.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:40:35.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:40:35.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:35 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:40:36.522 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T15:40:31.136454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T15:40:36.522 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-04T15:40:36.763 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:40:36.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:36 smithi080 ceph-mon[27564]: Detected new or changed devices on smithi087 2024-03-04T15:40:36.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:36 smithi080 ceph-mon[27564]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:40:36.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:36 smithi080 ceph-mon[27564]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:40:36.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:36 smithi080 ceph-mon[27564]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:40:36.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:36 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:36.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:36 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:36.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:36 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:36.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:36 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:36.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:36 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:36.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:36 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:36.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:36 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:36.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:36 smithi080 ceph-mon[27564]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:40:36.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:36 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:36.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:36 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/2796142862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:40:36.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:36 smithi005 ceph-mon[25858]: Detected new or changed devices on smithi087 2024-03-04T15:40:36.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:36 smithi005 ceph-mon[25858]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:40:36.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:36 smithi005 ceph-mon[25858]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:40:36.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:36 smithi005 ceph-mon[25858]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:40:36.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:36 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:36.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:36 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:36.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:36 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:36.884 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:36 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:36.885 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:36 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:36.885 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:36 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:36.885 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:36 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:36.885 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:36 smithi005 ceph-mon[25858]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:40:36.885 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:36 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:36.885 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:36 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2796142862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:40:37.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:36 smithi087 ceph-mon[27664]: Detected new or changed devices on smithi087 2024-03-04T15:40:37.008 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:36 smithi087 ceph-mon[27664]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:40:37.008 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:36 smithi087 ceph-mon[27664]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:40:37.008 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:36 smithi087 ceph-mon[27664]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:40:37.008 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:36 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:37.008 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:36 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:37.008 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:36 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:37.008 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:36 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:37.008 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:36 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:37.008 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:36 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:37.008 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:36 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:37.008 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:36 smithi087 ceph-mon[27664]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:40:37.008 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:36 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:37.009 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:36 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/2796142862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:40:38.604 INFO:teuthology.orchestra.run.smithi005.stdout:pg_num: 1 2024-03-04T15:40:38.911 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:38 smithi005 ceph-mon[25858]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:40:38.912 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:38 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/3273479206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T15:40:38.936 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:38 smithi087 ceph-mon[27664]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:40:38.936 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:38 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/3273479206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T15:40:39.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:38 smithi080 ceph-mon[27564]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:40:39.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:38 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/3273479206' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T15:40:39.230 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi005 2024-03-04T15:40:39.230 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi005=foo.a' 2024-03-04T15:40:39.465 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:40:40.883 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:40 smithi087 ceph-mon[27664]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 8.4 KiB/s rd, 449 KiB/s wr, 28 op/s 2024-03-04T15:40:40.898 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:40 smithi080 ceph-mon[27564]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 8.4 KiB/s rd, 449 KiB/s wr, 28 op/s 2024-03-04T15:40:40.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:40 smithi005 ceph-mon[25858]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 8.4 KiB/s rd, 449 KiB/s wr, 28 op/s 2024-03-04T15:40:41.340 INFO:teuthology.orchestra.run.smithi087.stdout:Scheduled rgw.foo.a update... 2024-03-04T15:40:42.070 DEBUG:teuthology.orchestra.run.smithi005:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@rgw.foo.a.service 2024-03-04T15:40:42.073 INFO:tasks.cephadm:Adding prometheus.a on smithi080 2024-03-04T15:40:42.073 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph orch apply prometheus '1;smithi080=a' 2024-03-04T15:40:42.307 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:40:42.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:42 smithi005 ceph-mon[25858]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi005=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:42.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:42 smithi005 ceph-mon[25858]: Saving service rgw.foo.a spec with placement smithi005=foo.a;count:1 2024-03-04T15:40:42.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:42 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:42.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:42 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:42.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:42 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:42.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:42 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:42 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:42 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T15:40:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:42 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T15:40:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:42 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:42 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:42 smithi005 ceph-mon[25858]: Deploying daemon rgw.foo.a on smithi005 2024-03-04T15:40:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:42 smithi005 ceph-mon[25858]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 6.4 KiB/s rd, 342 KiB/s wr, 21 op/s 2024-03-04T15:40:42.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:42 smithi087 ceph-mon[27664]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi005=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:42.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:42 smithi087 ceph-mon[27664]: Saving service rgw.foo.a spec with placement smithi005=foo.a;count:1 2024-03-04T15:40:42.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:42 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:42.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:42 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:42.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:42 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:42.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:42 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:42.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:42 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:42.591 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:42 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T15:40:42.591 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:42 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T15:40:42.591 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:42 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:42.591 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:42 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:42.591 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:42 smithi087 ceph-mon[27664]: Deploying daemon rgw.foo.a on smithi005 2024-03-04T15:40:42.591 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:42 smithi087 ceph-mon[27664]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 6.4 KiB/s rd, 342 KiB/s wr, 21 op/s 2024-03-04T15:40:42.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:42 smithi080 ceph-mon[27564]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi005=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:42.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:42 smithi080 ceph-mon[27564]: Saving service rgw.foo.a spec with placement smithi005=foo.a;count:1 2024-03-04T15:40:42.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:42 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:42.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:42 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:42.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:42 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:42.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:42 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:42.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:42 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:42.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:42 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T15:40:42.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:42 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T15:40:42.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:42 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:42.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:42 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:42.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:42 smithi080 ceph-mon[27564]: Deploying daemon rgw.foo.a on smithi005 2024-03-04T15:40:42.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:42 smithi080 ceph-mon[27564]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 6.4 KiB/s rd, 342 KiB/s wr, 21 op/s 2024-03-04T15:40:43.664 INFO:journalctl@ceph.rgw.foo.a.smithi005.stdout:Mar 04 15:40:43 smithi005 systemd[1]: Starting Ceph rgw.foo.a for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:40:43.931 INFO:journalctl@ceph.rgw.foo.a.smithi005.stdout:Mar 04 15:40:43 smithi005 podman[41499]: 2024-03-04 15:40:43.565467132 +0000 UTC m=+0.017722500 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:40:43.931 INFO:journalctl@ceph.rgw.foo.a.smithi005.stdout:Mar 04 15:40:43 smithi005 podman[41499]: 2024-03-04 15:40:43.702139467 +0000 UTC m=+0.154394816 container create 3c0a150b7041738a3439e29efb50e2f0e17f304302f17b624f95206ac0116c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-rgw-foo-a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-04T15:40:43.931 INFO:journalctl@ceph.rgw.foo.a.smithi005.stdout:Mar 04 15:40:43 smithi005 podman[41499]: 2024-03-04 15:40:43.896487992 +0000 UTC m=+0.348743342 container init 3c0a150b7041738a3439e29efb50e2f0e17f304302f17b624f95206ac0116c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD) 2024-03-04T15:40:43.931 INFO:journalctl@ceph.rgw.foo.a.smithi005.stdout:Mar 04 15:40:43 smithi005 podman[41499]: 2024-03-04 15:40:43.901816749 +0000 UTC m=+0.354072098 container start 3c0a150b7041738a3439e29efb50e2f0e17f304302f17b624f95206ac0116c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2) 2024-03-04T15:40:44.190 INFO:teuthology.orchestra.run.smithi087.stdout:Scheduled prometheus update... 2024-03-04T15:40:44.431 INFO:journalctl@ceph.rgw.foo.a.smithi005.stdout:Mar 04 15:40:43 smithi005 bash[41499]: 3c0a150b7041738a3439e29efb50e2f0e17f304302f17b624f95206ac0116c68 2024-03-04T15:40:44.431 INFO:journalctl@ceph.rgw.foo.a.smithi005.stdout:Mar 04 15:40:43 smithi005 systemd[1]: Started Ceph rgw.foo.a for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:40:44.746 DEBUG:teuthology.orchestra.run.smithi080:prometheus.a> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@prometheus.a.service 2024-03-04T15:40:44.749 INFO:tasks.cephadm:Adding node-exporter.a on smithi005 2024-03-04T15:40:44.749 INFO:tasks.cephadm:Adding node-exporter.b on smithi080 2024-03-04T15:40:44.749 INFO:tasks.cephadm:Adding node-exporter.c on smithi087 2024-03-04T15:40:44.749 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph orch apply node-exporter '3;smithi005=a;smithi080=b;smithi087=c' 2024-03-04T15:40:44.979 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:40:45.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:44 smithi087 ceph-mon[27664]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 290 KiB/s wr, 18 op/s 2024-03-04T15:40:45.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:44 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:44 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:44 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:44 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:44 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:44 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:44 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:45.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:44 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:45.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:44 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:45.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:44 smithi080 ceph-mon[27564]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 290 KiB/s wr, 18 op/s 2024-03-04T15:40:45.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:44 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:44 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:44 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:44 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:44 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:44 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:44 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:45.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:44 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:45.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:44 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:45.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:44 smithi005 ceph-mon[25858]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 290 KiB/s wr, 18 op/s 2024-03-04T15:40:45.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:44 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:44 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:44 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:44 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:44 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:44 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:45.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:44 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:40:45.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:44 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:40:45.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:44 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:40:45.817 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:45 smithi005 ceph-mon[25858]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi080=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:45.817 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:45 smithi005 ceph-mon[25858]: Saving service prometheus spec with placement smithi080=a;count:1 2024-03-04T15:40:45.817 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:45 smithi005 ceph-mon[25858]: Saving service rgw.foo.a spec with placement smithi005=foo.a;count:1 2024-03-04T15:40:45.817 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:45 smithi005 ceph-mon[25858]: Checking dashboard <-> RGW credentials 2024-03-04T15:40:45.817 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:45 smithi005 ceph-mon[25858]: osdmap e23: 3 total, 3 up, 3 in 2024-03-04T15:40:45.817 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:45 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/1323785867' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-04T15:40:45.818 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:45 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/4099021099' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-04T15:40:46.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:45 smithi087 ceph-mon[27664]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi080=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:46.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:45 smithi087 ceph-mon[27664]: Saving service prometheus spec with placement smithi080=a;count:1 2024-03-04T15:40:46.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:45 smithi087 ceph-mon[27664]: Saving service rgw.foo.a spec with placement smithi005=foo.a;count:1 2024-03-04T15:40:46.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:45 smithi087 ceph-mon[27664]: Checking dashboard <-> RGW credentials 2024-03-04T15:40:46.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:45 smithi087 ceph-mon[27664]: osdmap e23: 3 total, 3 up, 3 in 2024-03-04T15:40:46.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:45 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/1323785867' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-04T15:40:46.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:45 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/4099021099' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-04T15:40:46.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:45 smithi080 ceph-mon[27564]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi080=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:46.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:45 smithi080 ceph-mon[27564]: Saving service prometheus spec with placement smithi080=a;count:1 2024-03-04T15:40:46.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:45 smithi080 ceph-mon[27564]: Saving service rgw.foo.a spec with placement smithi005=foo.a;count:1 2024-03-04T15:40:46.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:45 smithi080 ceph-mon[27564]: Checking dashboard <-> RGW credentials 2024-03-04T15:40:46.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:45 smithi080 ceph-mon[27564]: osdmap e23: 3 total, 3 up, 3 in 2024-03-04T15:40:46.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:45 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/1323785867' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-04T15:40:46.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:45 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/4099021099' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-04T15:40:46.786 INFO:teuthology.orchestra.run.smithi087.stdout:Scheduled node-exporter update... 2024-03-04T15:40:46.828 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:46 smithi005 ceph-mon[25858]: pgmap v102: 33 pgs: 14 creating+peering, 1 active+clean, 18 unknown; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 5.4 KiB/s rd, 287 KiB/s wr, 18 op/s 2024-03-04T15:40:46.828 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:46 smithi005 ceph-mon[25858]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T15:40:46.828 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:46 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/1323785867' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-04T15:40:46.828 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:46 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/4099021099' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-04T15:40:46.829 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:46 smithi005 ceph-mon[25858]: osdmap e24: 3 total, 3 up, 3 in 2024-03-04T15:40:47.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:46 smithi087 ceph-mon[27664]: pgmap v102: 33 pgs: 14 creating+peering, 1 active+clean, 18 unknown; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 5.4 KiB/s rd, 287 KiB/s wr, 18 op/s 2024-03-04T15:40:47.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:46 smithi087 ceph-mon[27664]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T15:40:47.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:46 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/1323785867' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-04T15:40:47.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:46 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/4099021099' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-04T15:40:47.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:46 smithi087 ceph-mon[27664]: osdmap e24: 3 total, 3 up, 3 in 2024-03-04T15:40:47.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:46 smithi080 ceph-mon[27564]: pgmap v102: 33 pgs: 14 creating+peering, 1 active+clean, 18 unknown; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 5.4 KiB/s rd, 287 KiB/s wr, 18 op/s 2024-03-04T15:40:47.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:46 smithi080 ceph-mon[27564]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T15:40:47.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:46 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/1323785867' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-04T15:40:47.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:46 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/4099021099' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-04T15:40:47.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:46 smithi080 ceph-mon[27564]: osdmap e24: 3 total, 3 up, 3 in 2024-03-04T15:40:47.406 DEBUG:teuthology.orchestra.run.smithi005:node-exporter.a> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@node-exporter.a.service 2024-03-04T15:40:47.409 DEBUG:teuthology.orchestra.run.smithi080:node-exporter.b> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@node-exporter.b.service 2024-03-04T15:40:47.411 DEBUG:teuthology.orchestra.run.smithi087:node-exporter.c> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@node-exporter.c.service 2024-03-04T15:40:47.414 INFO:tasks.cephadm:Adding alertmanager.a on smithi005 2024-03-04T15:40:47.414 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph orch apply alertmanager '1;smithi005=a' 2024-03-04T15:40:47.687 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:40:48.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:47 smithi087 ceph-mon[27664]: osdmap e25: 3 total, 3 up, 3 in 2024-03-04T15:40:48.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:47 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T15:40:48.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:47 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T15:40:48.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:47 smithi087 ceph-mon[27664]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi005=a;smithi080=b;smithi087=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:48.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:47 smithi087 ceph-mon[27664]: Saving service node-exporter spec with placement smithi005=a;smithi080=b;smithi087=c;count:3 2024-03-04T15:40:48.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:47 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:48.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:47 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:48.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:47 smithi080 ceph-mon[27564]: osdmap e25: 3 total, 3 up, 3 in 2024-03-04T15:40:48.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:47 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T15:40:48.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:47 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T15:40:48.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:47 smithi080 ceph-mon[27564]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi005=a;smithi080=b;smithi087=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:48.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:47 smithi080 ceph-mon[27564]: Saving service node-exporter spec with placement smithi005=a;smithi080=b;smithi087=c;count:3 2024-03-04T15:40:48.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:47 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:48.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:47 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:48.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:47 smithi005 ceph-mon[25858]: osdmap e25: 3 total, 3 up, 3 in 2024-03-04T15:40:48.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:47 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T15:40:48.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:47 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T15:40:48.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:47 smithi005 ceph-mon[25858]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi005=a;smithi080=b;smithi087=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:48.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:47 smithi005 ceph-mon[25858]: Saving service node-exporter spec with placement smithi005=a;smithi080=b;smithi087=c;count:3 2024-03-04T15:40:48.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:47 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:48.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:47 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:48.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:48 smithi005 ceph-mon[25858]: pgmap v105: 65 pgs: 26 creating+peering, 9 active+clean, 30 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 341 B/s wr, 0 op/s 2024-03-04T15:40:48.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:48 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T15:40:48.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:48 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T15:40:48.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:48 smithi005 ceph-mon[25858]: osdmap e26: 3 total, 3 up, 3 in 2024-03-04T15:40:48.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:48 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:48.995 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:48 smithi087 ceph-mon[27664]: pgmap v105: 65 pgs: 26 creating+peering, 9 active+clean, 30 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 341 B/s wr, 0 op/s 2024-03-04T15:40:48.995 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:48 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T15:40:48.995 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:48 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T15:40:48.995 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:48 smithi087 ceph-mon[27664]: osdmap e26: 3 total, 3 up, 3 in 2024-03-04T15:40:48.995 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:48 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:49.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:48 smithi080 ceph-mon[27564]: pgmap v105: 65 pgs: 26 creating+peering, 9 active+clean, 30 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 341 B/s wr, 0 op/s 2024-03-04T15:40:49.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:48 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T15:40:49.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:48 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T15:40:49.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:48 smithi080 ceph-mon[27564]: osdmap e26: 3 total, 3 up, 3 in 2024-03-04T15:40:49.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:48 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:49.458 INFO:teuthology.orchestra.run.smithi087.stdout:Scheduled alertmanager update... 2024-03-04T15:40:50.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:49 smithi087 ceph-mon[27664]: osdmap e27: 3 total, 3 up, 3 in 2024-03-04T15:40:50.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:49 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T15:40:50.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:49 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T15:40:50.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:49 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:50.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:49 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:50.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:49 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:50.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:49 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:50.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:49 smithi080 ceph-mon[27564]: osdmap e27: 3 total, 3 up, 3 in 2024-03-04T15:40:50.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:49 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T15:40:50.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:49 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T15:40:50.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:49 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:50.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:49 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:50.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:49 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:50.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:49 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:50.099 DEBUG:teuthology.orchestra.run.smithi005:alertmanager.a> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@alertmanager.a.service 2024-03-04T15:40:50.102 INFO:tasks.cephadm:Adding grafana.a on smithi080 2024-03-04T15:40:50.102 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph orch apply grafana '1;smithi080=a' 2024-03-04T15:40:50.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:49 smithi005 ceph-mon[25858]: osdmap e27: 3 total, 3 up, 3 in 2024-03-04T15:40:50.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:49 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T15:40:50.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:49 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T15:40:50.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:49 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:50.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:49 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:50.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:49 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:50.125 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:49 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:50.333 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:40:51.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:50 smithi087 ceph-mon[27664]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi005=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:51.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:50 smithi087 ceph-mon[27664]: Saving service alertmanager spec with placement smithi005=a;count:1 2024-03-04T15:40:51.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:50 smithi087 ceph-mon[27664]: pgmap v108: 97 pgs: 26 creating+peering, 30 active+clean, 41 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 511 B/s wr, 0 op/s 2024-03-04T15:40:51.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:50 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T15:40:51.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:50 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T15:40:51.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:50 smithi087 ceph-mon[27664]: osdmap e28: 3 total, 3 up, 3 in 2024-03-04T15:40:51.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:50 smithi080 ceph-mon[27564]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi005=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:51.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:50 smithi080 ceph-mon[27564]: Saving service alertmanager spec with placement smithi005=a;count:1 2024-03-04T15:40:51.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:50 smithi080 ceph-mon[27564]: pgmap v108: 97 pgs: 26 creating+peering, 30 active+clean, 41 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 511 B/s wr, 0 op/s 2024-03-04T15:40:51.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:50 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T15:40:51.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:50 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T15:40:51.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:50 smithi080 ceph-mon[27564]: osdmap e28: 3 total, 3 up, 3 in 2024-03-04T15:40:51.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:50 smithi005 ceph-mon[25858]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi005=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:51.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:50 smithi005 ceph-mon[25858]: Saving service alertmanager spec with placement smithi005=a;count:1 2024-03-04T15:40:51.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:50 smithi005 ceph-mon[25858]: pgmap v108: 97 pgs: 26 creating+peering, 30 active+clean, 41 unknown; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 511 B/s wr, 0 op/s 2024-03-04T15:40:51.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:50 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T15:40:51.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:50 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T15:40:51.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:50 smithi005 ceph-mon[25858]: osdmap e28: 3 total, 3 up, 3 in 2024-03-04T15:40:51.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:51 smithi080 ceph-mon[27564]: osdmap e29: 3 total, 3 up, 3 in 2024-03-04T15:40:51.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:51 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T15:40:51.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:51 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T15:40:51.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:51 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T15:40:52.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:51 smithi087 ceph-mon[27664]: osdmap e29: 3 total, 3 up, 3 in 2024-03-04T15:40:52.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:51 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T15:40:52.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:51 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T15:40:52.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:51 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T15:40:52.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:51 smithi005 ceph-mon[25858]: osdmap e29: 3 total, 3 up, 3 in 2024-03-04T15:40:52.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:51 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T15:40:52.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:51 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T15:40:52.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:51 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T15:40:52.190 INFO:teuthology.orchestra.run.smithi087.stdout:Scheduled grafana update... 2024-03-04T15:40:52.840 DEBUG:teuthology.orchestra.run.smithi080:grafana.a> sudo journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@grafana.a.service 2024-03-04T15:40:52.843 INFO:tasks.cephadm:Setting up client nodes... 2024-03-04T15:40:52.843 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-04T15:40:53.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:52 smithi087 ceph-mon[27664]: pgmap v111: 129 pgs: 12 creating+peering, 65 active+clean, 52 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-04T15:40:53.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:52 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T15:40:53.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:52 smithi087 ceph-mon[27664]: osdmap e30: 3 total, 3 up, 3 in 2024-03-04T15:40:53.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:52 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T15:40:53.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:52 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T15:40:53.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:52 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:53.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:52 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T15:40:53.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:52 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T15:40:53.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:52 smithi087 ceph-mon[27664]: osdmap e31: 3 total, 3 up, 3 in 2024-03-04T15:40:53.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:52 smithi080 ceph-mon[27564]: pgmap v111: 129 pgs: 12 creating+peering, 65 active+clean, 52 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-04T15:40:53.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:52 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T15:40:53.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:52 smithi080 ceph-mon[27564]: osdmap e30: 3 total, 3 up, 3 in 2024-03-04T15:40:53.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:52 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T15:40:53.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:52 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T15:40:53.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:52 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:53.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:52 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T15:40:53.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:52 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T15:40:53.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:52 smithi080 ceph-mon[27564]: osdmap e31: 3 total, 3 up, 3 in 2024-03-04T15:40:53.140 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:40:53.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:52 smithi005 ceph-mon[25858]: pgmap v111: 129 pgs: 12 creating+peering, 65 active+clean, 52 unknown; 578 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-04T15:40:53.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:52 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T15:40:53.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:52 smithi005 ceph-mon[25858]: osdmap e30: 3 total, 3 up, 3 in 2024-03-04T15:40:53.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:52 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T15:40:53.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:52 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T15:40:53.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:52 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:53.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:52 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/411910337' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T15:40:53.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:52 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2206826432' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T15:40:53.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:52 smithi005 ceph-mon[25858]: osdmap e31: 3 total, 3 up, 3 in 2024-03-04T15:40:53.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:53 smithi005 ceph-mon[25858]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi080=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:53.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:53 smithi005 ceph-mon[25858]: Saving service grafana spec with placement smithi080=a;count:1 2024-03-04T15:40:53.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:53 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:53.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:53 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:53.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:53 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:53.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:53 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:53.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:53 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:53.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:53 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:54.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:53 smithi087 ceph-mon[27664]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi080=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:54.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:53 smithi087 ceph-mon[27664]: Saving service grafana spec with placement smithi080=a;count:1 2024-03-04T15:40:54.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:53 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:54.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:53 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:54.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:53 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:54.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:53 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:54.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:53 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:54.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:53 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:54.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:53 smithi080 ceph-mon[27564]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi080=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:40:54.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:53 smithi080 ceph-mon[27564]: Saving service grafana spec with placement smithi080=a;count:1 2024-03-04T15:40:54.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:53 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:54.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:53 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:54.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:53 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:54.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:53 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:54.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:53 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:54.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:53 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:54.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:54 smithi005 ceph-mon[25858]: Deploying daemon prometheus.a on smithi080 2024-03-04T15:40:54.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:54 smithi005 ceph-mon[25858]: pgmap v114: 129 pgs: 120 active+clean, 9 unknown; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 45 KiB/s rd, 4.0 KiB/s wr, 107 op/s 2024-03-04T15:40:54.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:54 smithi005 ceph-mon[25858]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T15:40:54.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:54 smithi005 ceph-mon[25858]: Cluster is now healthy 2024-03-04T15:40:55.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:54 smithi087 ceph-mon[27664]: Deploying daemon prometheus.a on smithi080 2024-03-04T15:40:55.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:54 smithi087 ceph-mon[27664]: pgmap v114: 129 pgs: 120 active+clean, 9 unknown; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 45 KiB/s rd, 4.0 KiB/s wr, 107 op/s 2024-03-04T15:40:55.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:54 smithi087 ceph-mon[27664]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T15:40:55.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:54 smithi087 ceph-mon[27664]: Cluster is now healthy 2024-03-04T15:40:55.058 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:54 smithi080 ceph-mon[27564]: Deploying daemon prometheus.a on smithi080 2024-03-04T15:40:55.058 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:54 smithi080 ceph-mon[27564]: pgmap v114: 129 pgs: 120 active+clean, 9 unknown; 579 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 45 KiB/s rd, 4.0 KiB/s wr, 107 op/s 2024-03-04T15:40:55.058 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:54 smithi080 ceph-mon[27564]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T15:40:55.058 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:54 smithi080 ceph-mon[27564]: Cluster is now healthy 2024-03-04T15:40:55.233 INFO:teuthology.orchestra.run.smithi005.stdout:[client.0] 2024-03-04T15:40:55.233 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCH6+VlzciuDRAAeyl8l4EcnSthFZpFaYPsNA== 2024-03-04T15:40:55.802 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:55 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/898979793' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:40:55.803 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:55 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/898979793' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:40:55.803 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:55.803 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:55.803 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:55 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:55.829 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-04T15:40:55.830 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-04T15:40:55.830 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-04T15:40:55.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:55 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/898979793' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:40:55.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:55 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/898979793' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:40:55.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:55.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:55.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:55 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:55.872 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-04T15:40:56.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:55 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/898979793' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:40:56.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:55 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/898979793' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:40:56.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:56.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:56.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:55 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:40:56.572 INFO:teuthology.orchestra.run.smithi080.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T15:40:57.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:56 smithi087 ceph-mon[27664]: pgmap v115: 129 pgs: 129 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 54 KiB/s rd, 5.0 KiB/s wr, 128 op/s 2024-03-04T15:40:57.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:56 smithi080 ceph-mon[27564]: pgmap v115: 129 pgs: 129 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 54 KiB/s rd, 5.0 KiB/s wr, 128 op/s 2024-03-04T15:40:57.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:56 smithi005 ceph-mon[25858]: pgmap v115: 129 pgs: 129 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 54 KiB/s rd, 5.0 KiB/s wr, 128 op/s 2024-03-04T15:40:59.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:58 smithi087 ceph-mon[27664]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 68 KiB/s rd, 5.3 KiB/s wr, 164 op/s 2024-03-04T15:40:59.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:58 smithi080 ceph-mon[27564]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 68 KiB/s rd, 5.3 KiB/s wr, 164 op/s 2024-03-04T15:40:59.132 INFO:teuthology.orchestra.run.smithi080.stdout:[client.1] 2024-03-04T15:40:59.133 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQCL6+VlUc2TBxAA/uDjDeVsJFJsSNqGr6a9ng== 2024-03-04T15:40:59.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:58 smithi005 ceph-mon[25858]: pgmap v116: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 68 KiB/s rd, 5.3 KiB/s wr, 164 op/s 2024-03-04T15:41:00.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:59 smithi087 ceph-mon[27664]: from='client.? 172.21.15.80:0/1824480659' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:41:00.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:59 smithi087 ceph-mon[27664]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:41:00.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:40:59 smithi087 ceph-mon[27664]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:41:00.072 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:59 smithi080 ceph-mon[27564]: from='client.? 172.21.15.80:0/1824480659' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:41:00.072 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:59 smithi080 ceph-mon[27564]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:41:00.072 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:40:59 smithi080 ceph-mon[27564]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:41:00.097 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-04T15:41:00.097 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-04T15:41:00.097 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-04T15:41:00.140 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-04T15:41:00.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:59 smithi005 ceph-mon[25858]: from='client.? 172.21.15.80:0/1824480659' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:41:00.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:59 smithi005 ceph-mon[25858]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:41:00.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:40:59 smithi005 ceph-mon[25858]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:41:00.388 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:41:01.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:00 smithi087 ceph-mon[27664]: pgmap v117: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-04T15:41:01.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:00 smithi080 ceph-mon[27564]: pgmap v117: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-04T15:41:01.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:00 smithi005 ceph-mon[25858]: pgmap v117: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-04T15:41:02.123 INFO:teuthology.orchestra.run.smithi087.stdout:[client.2] 2024-03-04T15:41:02.123 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCO6+VljVEABxAAzZLje/Kbu8EC6CMkHD37MA== 2024-03-04T15:41:02.796 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-04T15:41:02.796 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-04T15:41:02.797 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-04T15:41:02.838 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-04T15:41:02.838 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-04T15:41:02.839 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph mgr dump --format=json 2024-03-04T15:41:03.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:02 smithi087 ceph-mon[27664]: pgmap v118: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 47 KiB/s rd, 3.7 KiB/s wr, 114 op/s 2024-03-04T15:41:03.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:02 smithi087 ceph-mon[27664]: from='client.? 172.21.15.87:0/853496479' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:41:03.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:02 smithi087 ceph-mon[27664]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:41:03.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:02 smithi087 ceph-mon[27664]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:41:03.087 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:03.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:02 smithi080 ceph-mon[27564]: pgmap v118: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 47 KiB/s rd, 3.7 KiB/s wr, 114 op/s 2024-03-04T15:41:03.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:02 smithi080 ceph-mon[27564]: from='client.? 172.21.15.87:0/853496479' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:41:03.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:02 smithi080 ceph-mon[27564]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:41:03.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:02 smithi080 ceph-mon[27564]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:41:03.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:02 smithi005 ceph-mon[25858]: pgmap v118: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 47 KiB/s rd, 3.7 KiB/s wr, 114 op/s 2024-03-04T15:41:03.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:02 smithi005 ceph-mon[25858]: from='client.? 172.21.15.87:0/853496479' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:41:03.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:02 smithi005 ceph-mon[25858]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:41:03.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:02 smithi005 ceph-mon[25858]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:41:04.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:04 smithi005 ceph-mon[25858]: pgmap v119: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 43 KiB/s rd, 3.2 KiB/s wr, 103 op/s 2024-03-04T15:41:05.009 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:04 smithi080 ceph-mon[27564]: pgmap v119: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 43 KiB/s rd, 3.2 KiB/s wr, 103 op/s 2024-03-04T15:41:05.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:04 smithi087 ceph-mon[27664]: pgmap v119: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 43 KiB/s rd, 3.2 KiB/s wr, 103 op/s 2024-03-04T15:41:05.112 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:41:05.796 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":1931015659},{"type":"v1","addr":"172.21.15.5:6801","nonce":1931015659}]},"active_addr":"172.21.15.5:6801/1931015659","active_change":"2024-03-04T15:37:56.775923+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.5:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":1649031538}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":3311714078}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":1561918965}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":2131603651}]}]} 2024-03-04T15:41:05.802 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-04T15:41:05.802 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-04T15:41:05.802 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T15:41:06.011 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:05 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2771542104' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T15:41:06.012 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:05 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/2771542104' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T15:41:06.012 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:05 smithi080 systemd[1]: Starting Ceph prometheus.a for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:41:06.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:05 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/2771542104' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T15:41:06.280 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 podman[35982]: 2024-03-04 15:41:06.010276801 +0000 UTC m=+0.112044583 container create 713fb4e36d740818b3492c79e68b212f9105358495ce0a23c3ae54f188921b73 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T15:41:06.280 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 podman[35982]: 2024-03-04 15:41:05.916680561 +0000 UTC m=+0.018448358 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-04T15:41:06.578 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 podman[35982]: 2024-03-04 15:41:06.279071468 +0000 UTC m=+0.380839253 container init 713fb4e36d740818b3492c79e68b212f9105358495ce0a23c3ae54f188921b73 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T15:41:06.579 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 podman[35982]: 2024-03-04 15:41:06.28425425 +0000 UTC m=+0.386022033 container start 713fb4e36d740818b3492c79e68b212f9105358495ce0a23c3ae54f188921b73 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T15:41:06.579 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.307Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-04T15:41:06.579 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.307Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-04T15:41:06.579 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.307Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi080 (none))" 2024-03-04T15:41:06.579 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.307Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-04T15:41:06.579 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.307Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-04T15:41:06.579 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.308Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-04T15:41:06.579 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.309Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-04T15:41:06.579 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.309Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-04T15:41:06.579 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.310Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-04T15:41:06.579 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.312Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-04T15:41:06.579 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.312Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.608µs 2024-03-04T15:41:06.579 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.312Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-04T15:41:06.580 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.312Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-04T15:41:06.580 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.312Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=16.758µs wal_replay_duration=195.242µs wbl_replay_duration=88ns total_replay_duration=229.116µs 2024-03-04T15:41:06.580 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.313Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-04T15:41:06.580 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.313Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-04T15:41:06.580 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.313Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-04T15:41:06.580 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.332Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.796516ms db_storage=727ns remote_storage=1.53µs web_handler=373ns query_engine=501ns scrape=189.184µs scrape_sd=70.649µs notify=880ns notify_sd=2.023µs rules=18.361658ms tracing=5.163µs 2024-03-04T15:41:06.580 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.332Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-04T15:41:06.580 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:41:06.332Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-04T15:41:06.580 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 bash[35982]: 713fb4e36d740818b3492c79e68b212f9105358495ce0a23c3ae54f188921b73 2024-03-04T15:41:06.580 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:41:06 smithi080 systemd[1]: Started Ceph prometheus.a for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:41:06.585 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:06.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-mon[27564]: pgmap v120: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 25 KiB/s rd, 2.0 KiB/s wr, 59 op/s 2024-03-04T15:41:06.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:41:06.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:41:06.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:41:06.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:06 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-04T15:41:07.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:06 smithi005 ceph-mon[25858]: pgmap v120: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 25 KiB/s rd, 2.0 KiB/s wr, 59 op/s 2024-03-04T15:41:07.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:41:07.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:41:07.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:41:07.173 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:06 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-04T15:41:07.178 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:06 smithi087 ceph-mon[27664]: pgmap v120: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 25 KiB/s rd, 2.0 KiB/s wr, 59 op/s 2024-03-04T15:41:07.179 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:41:07.179 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:41:07.179 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:41:07.179 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:06 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-04T15:41:07.760 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:07 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: ignoring --setuser ceph since I am not root 2024-03-04T15:41:07.760 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:07 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: ignoring --setgroup ceph since I am not root 2024-03-04T15:41:07.760 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:07 smithi087 ceph-mgr[30191]: -- 172.21.15.87:0/145819222 <== mon.1 v2:172.21.15.87:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5628ea1365a0 con 0x5628ea10ac00 2024-03-04T15:41:07.789 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:07 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: ignoring --setuser ceph since I am not root 2024-03-04T15:41:07.789 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:07 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: ignoring --setgroup ceph since I am not root 2024-03-04T15:41:07.810 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: ignoring --setuser ceph since I am not root 2024-03-04T15:41:07.810 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: ignoring --setgroup ceph since I am not root 2024-03-04T15:41:08.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:07 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:41:08.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:07 smithi087 ceph-mon[27664]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-04T15:41:08.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:07 smithi087 ceph-mon[27664]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-04T15:41:08.044 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:07 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:41:08.044 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:07 smithi080 ceph-mon[27564]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-04T15:41:08.044 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:07 smithi080 ceph-mon[27564]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-04T15:41:08.072 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:07 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' 2024-03-04T15:41:08.072 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:07 smithi005 ceph-mon[25858]: from='mgr.14152 172.21.15.5:0/3095760261' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-04T15:41:08.072 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:07 smithi005 ceph-mon[25858]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-04T15:41:08.346 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:08 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:08.042+0000 7f152a5c3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:41:08.346 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:08 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:08.136+0000 7f152a5c3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:41:08.388 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:08 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:08.069+0000 7fcea30a4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:41:08.388 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:08 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:08.169+0000 7fcea30a4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:41:08.527 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:08 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:08.070+0000 7f27eb9a8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:41:08.527 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:08 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:08.165+0000 7f27eb9a8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:41:08.999 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:08 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:08.900+0000 7fcea30a4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:41:09.027 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:08 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:08.805+0000 7f27eb9a8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:41:09.027 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:08 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:08.897+0000 7f27eb9a8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:41:09.090 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:08 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:08.765+0000 7f152a5c3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:41:09.090 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:08 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:08.856+0000 7f152a5c3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:41:09.090 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:08 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:08.994+0000 7f152a5c3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:41:09.090 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:09 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:09.087+0000 7f152a5c3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:41:09.301 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:08 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:08.996+0000 7fcea30a4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:41:09.301 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:09 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:09.141+0000 7fcea30a4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:41:09.301 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:09 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:09.239+0000 7fcea30a4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:41:09.346 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:09 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:09.171+0000 7f152a5c3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:41:09.473 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:09 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:09.036+0000 7f27eb9a8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:41:09.474 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:09 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:09.131+0000 7f27eb9a8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:41:09.474 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:09 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:09.214+0000 7f27eb9a8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:41:09.474 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:09 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:09.472+0000 7f27eb9a8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:41:09.577 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:09 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:09.326+0000 7fcea30a4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:41:09.718 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:09 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:09.430+0000 7f152a5c3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:41:09.719 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:09 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:09.538+0000 7f152a5c3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:41:09.719 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:09 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:09.623+0000 7f152a5c3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:41:09.719 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:09 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:09.716+0000 7f152a5c3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:41:09.762 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:09 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:09.580+0000 7f27eb9a8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:41:09.762 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:09 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:09.665+0000 7f27eb9a8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:41:09.762 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:09 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:09.760+0000 7f27eb9a8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:41:09.900 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:41:09.900 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":31,"fsid":"97c626fc-da3b-11ee-95c6-87774f69a715","created":"2024-03-04T15:37:02.677902+0000","modified":"2024-03-04T15:40:52.752673+0000","last_up_change":"2024-03-04T15:40:28.959524+0000","last_in_change":"2024-03-04T15:40:08.226352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:40:31.136454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-04T15:40:44.181345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-04T15:40:46.040784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-04T15:40:47.999682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-04T15:40:50.070369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e8ffc2cd-fb54-4a4f-b317-f543ca3cbbfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":3224525915},{"type":"v1","addr":"172.21.15.5:6803","nonce":3224525915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":3224525915},{"type":"v1","addr":"172.21.15.5:6805","nonce":3224525915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":3224525915},{"type":"v1","addr":"172.21.15.5:6809","nonce":3224525915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":3224525915},{"type":"v1","addr":"172.21.15.5:6807","nonce":3224525915}]},"public_addr":"172.21.15.5:6803/3224525915","cluster_addr":"172.21.15.5:6805/3224525915","heartbeat_back_addr":"172.21.15.5:6809/3224525915","heartbeat_front_addr":"172.21.15.5:6807/3224525915","state":["exists","up"]},{"osd":1,"uuid":"6f5f296e-1fb4-410f-90a2-809ffcc01724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":1635610381},{"type":"v1","addr":"172.21.15.80:6801","nonce":1635610381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":1635610381},{"type":"v1","addr":"172.21.15.80:6803","nonce":1635610381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":1635610381},{"type":"v1","addr":"172.21.15.80:6807","nonce":1635610381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":1635610381},{"type":"v1","addr":"172.21.15.80:6805","nonce":1635610381}]},"public_addr":"172.21.15.80:6801/1635610381","cluster_addr":"172.21.15.80:6803/1635610381","heartbeat_back_addr":"172.21.15.80:6807/1635610381","heartbeat_front_addr":"172.21.15.80:6805/1635610381","state":["exists","up"]},{"osd":2,"uuid":"6c4fe192-e1b9-4d9f-a93d-1b760ad76c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":609742830},{"type":"v1","addr":"172.21.15.87:6801","nonce":609742830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":609742830},{"type":"v1","addr":"172.21.15.87:6803","nonce":609742830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":609742830},{"type":"v1","addr":"172.21.15.87:6807","nonce":609742830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":609742830},{"type":"v1","addr":"172.21.15.87:6805","nonce":609742830}]},"public_addr":"172.21.15.87:6801/609742830","cluster_addr":"172.21.15.87:6803/609742830","heartbeat_back_addr":"172.21.15.87:6807/609742830","heartbeat_front_addr":"172.21.15.87:6805/609742830","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:39:34.316398+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:40:00.254436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:40:26.922481+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/1666734806":"2024-03-05T15:37:56.775816+0000","172.21.15.5:0/1558141873":"2024-03-05T15:37:56.775816+0000","172.21.15.5:0/2354875202":"2024-03-05T15:37:56.775816+0000","172.21.15.5:0/1979102031":"2024-03-05T15:37:29.460739+0000","172.21.15.5:0/3924784696":"2024-03-05T15:37:29.460739+0000","172.21.15.5:6800/3259340716":"2024-03-05T15:37:29.460739+0000","172.21.15.5:6801/4290743325":"2024-03-05T15:37:56.775816+0000","172.21.15.5:6801/3259340716":"2024-03-05T15:37:29.460739+0000","172.21.15.5:0/3674674128":"2024-03-05T15:37:29.460739+0000","172.21.15.5:6800/4290743325":"2024-03-05T15:37:56.775816+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T15:41:09.900 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:09 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:09.596+0000 7fcea30a4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:41:09.900 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:09 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:09.713+0000 7fcea30a4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:41:09.901 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:09 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:09.804+0000 7fcea30a4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:41:10.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:09 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/2736114800' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:41:10.027 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:09 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:09.932+0000 7f27eb9a8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:41:10.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:09 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/2736114800' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:41:10.096 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:09 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:09.891+0000 7f152a5c3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:41:10.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:09 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2736114800' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:41:10.181 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:09 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:09.902+0000 7fcea30a4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:41:10.181 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:10 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:10.084+0000 7fcea30a4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:41:10.533 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-04T15:41:10.533 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T15:41:10.808 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:11.277 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:10 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:10.882+0000 7f27eb9a8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:41:11.277 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:11 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:11.010+0000 7f27eb9a8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:41:11.292 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:10 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:10.875+0000 7f152a5c3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:41:11.292 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:11.008+0000 7f152a5c3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:41:11.335 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:11 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:11.151+0000 7fcea30a4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:41:11.335 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:11 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:11.291+0000 7fcea30a4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:41:11.596 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:11.348+0000 7f152a5c3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:41:11.596 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:11.436+0000 7f152a5c3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:41:11.640 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:11 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:11.357+0000 7f27eb9a8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:41:11.640 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:11 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:11.446+0000 7f27eb9a8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:41:11.640 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:11 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:11.638+0000 7f27eb9a8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:41:11.931 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:11 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:11.635+0000 7fcea30a4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:41:11.931 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:11 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:11.731+0000 7fcea30a4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:41:12.027 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:11 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:11.820+0000 7f27eb9a8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:41:12.095 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:11.631+0000 7f152a5c3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:41:12.095 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:11.813+0000 7f152a5c3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:41:12.258 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:11 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:11.935+0000 7fcea30a4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:41:12.258 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:12 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:12.131+0000 7fcea30a4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:41:12.346 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:12 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:12.093+0000 7f152a5c3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:41:12.357 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:12 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:12.098+0000 7f27eb9a8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:41:12.634 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:12 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:12.378+0000 7f152a5c3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:41:12.634 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:12 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:12.632+0000 7f152a5c3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:41:12.681 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:12 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:12.433+0000 7fcea30a4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:41:12.712 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:41:12.712 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":31,"fsid":"97c626fc-da3b-11ee-95c6-87774f69a715","created":"2024-03-04T15:37:02.677902+0000","modified":"2024-03-04T15:40:52.752673+0000","last_up_change":"2024-03-04T15:40:28.959524+0000","last_in_change":"2024-03-04T15:40:08.226352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:40:31.136454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-04T15:40:44.181345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-04T15:40:46.040784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-04T15:40:47.999682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-04T15:40:50.070369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e8ffc2cd-fb54-4a4f-b317-f543ca3cbbfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":3224525915},{"type":"v1","addr":"172.21.15.5:6803","nonce":3224525915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":3224525915},{"type":"v1","addr":"172.21.15.5:6805","nonce":3224525915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":3224525915},{"type":"v1","addr":"172.21.15.5:6809","nonce":3224525915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":3224525915},{"type":"v1","addr":"172.21.15.5:6807","nonce":3224525915}]},"public_addr":"172.21.15.5:6803/3224525915","cluster_addr":"172.21.15.5:6805/3224525915","heartbeat_back_addr":"172.21.15.5:6809/3224525915","heartbeat_front_addr":"172.21.15.5:6807/3224525915","state":["exists","up"]},{"osd":1,"uuid":"6f5f296e-1fb4-410f-90a2-809ffcc01724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":1635610381},{"type":"v1","addr":"172.21.15.80:6801","nonce":1635610381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":1635610381},{"type":"v1","addr":"172.21.15.80:6803","nonce":1635610381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":1635610381},{"type":"v1","addr":"172.21.15.80:6807","nonce":1635610381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":1635610381},{"type":"v1","addr":"172.21.15.80:6805","nonce":1635610381}]},"public_addr":"172.21.15.80:6801/1635610381","cluster_addr":"172.21.15.80:6803/1635610381","heartbeat_back_addr":"172.21.15.80:6807/1635610381","heartbeat_front_addr":"172.21.15.80:6805/1635610381","state":["exists","up"]},{"osd":2,"uuid":"6c4fe192-e1b9-4d9f-a93d-1b760ad76c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":609742830},{"type":"v1","addr":"172.21.15.87:6801","nonce":609742830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":609742830},{"type":"v1","addr":"172.21.15.87:6803","nonce":609742830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":609742830},{"type":"v1","addr":"172.21.15.87:6807","nonce":609742830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":609742830},{"type":"v1","addr":"172.21.15.87:6805","nonce":609742830}]},"public_addr":"172.21.15.87:6801/609742830","cluster_addr":"172.21.15.87:6803/609742830","heartbeat_back_addr":"172.21.15.87:6807/609742830","heartbeat_front_addr":"172.21.15.87:6805/609742830","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:39:34.316398+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:40:00.254436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:40:26.922481+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/1666734806":"2024-03-05T15:37:56.775816+0000","172.21.15.5:0/1558141873":"2024-03-05T15:37:56.775816+0000","172.21.15.5:0/2354875202":"2024-03-05T15:37:56.775816+0000","172.21.15.5:0/1979102031":"2024-03-05T15:37:29.460739+0000","172.21.15.5:0/3924784696":"2024-03-05T15:37:29.460739+0000","172.21.15.5:6800/3259340716":"2024-03-05T15:37:29.460739+0000","172.21.15.5:6801/4290743325":"2024-03-05T15:37:56.775816+0000","172.21.15.5:6801/3259340716":"2024-03-05T15:37:29.460739+0000","172.21.15.5:0/3674674128":"2024-03-05T15:37:29.460739+0000","172.21.15.5:6800/4290743325":"2024-03-05T15:37:56.775816+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T15:41:12.762 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:12 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:12.355+0000 7f27eb9a8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:41:12.762 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:12 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:12.587+0000 7f27eb9a8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:41:13.011 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:12 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/438288814' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:41:13.026 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:12 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/438288814' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:41:13.026 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:12 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:12.699+0000 7fcea30a4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:41:13.026 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:12 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:12.947+0000 7fcea30a4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:41:13.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:12 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/438288814' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:41:13.304 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-04T15:41:13.305 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-04T15:41:13.305 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-04T15:41:13.589 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:13.590 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:13.590 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:13.754 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:13 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:13.509+0000 7f152a5c3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:41:13.777 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:13 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:13.372+0000 7f27eb9a8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:41:13.971 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:13 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:13.832+0000 7fcea30a4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:41:14.273 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:13 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:13.951+0000 7f27eb9a8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:41:14.273 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:14 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:14.050+0000 7f27eb9a8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:41:14.273 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:14 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:14.271+0000 7f27eb9a8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:41:14.346 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:14 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:14.086+0000 7f152a5c3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:41:14.346 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:14 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:14.219+0000 7f152a5c3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:41:14.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:14 smithi087 ceph-mon[27664]: Standby manager daemon z restarted 2024-03-04T15:41:14.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:14 smithi087 ceph-mon[27664]: Standby manager daemon z started 2024-03-04T15:41:14.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:14 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.87:0/3148068781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T15:41:14.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:14 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.87:0/3148068781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:41:14.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:14 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.87:0/3148068781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T15:41:14.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:14 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.87:0/3148068781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:41:14.528 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:14 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: [04/Mar/2024:15:41:14] ENGINE Bus STARTING 2024-03-04T15:41:14.528 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:14 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: CherryPy Checker: 2024-03-04T15:41:14.528 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:14 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: The Application mounted at '' has an empty config. 2024-03-04T15:41:14.528 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:14 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: 2024-03-04T15:41:14.528 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:14 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: [04/Mar/2024:15:41:14] ENGINE Serving on http://:::9283 2024-03-04T15:41:14.529 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:41:14 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z[30113]: [04/Mar/2024:15:41:14] ENGINE Bus STARTED 2024-03-04T15:41:14.544 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:14 smithi005 ceph-mon[25858]: Standby manager daemon z restarted 2024-03-04T15:41:14.544 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:14 smithi005 ceph-mon[25858]: Standby manager daemon z started 2024-03-04T15:41:14.544 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:14 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.87:0/3148068781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T15:41:14.544 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:14 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.87:0/3148068781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:41:14.544 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:14 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.87:0/3148068781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T15:41:14.544 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:14 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.87:0/3148068781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:41:14.544 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:14 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:14.432+0000 7fcea30a4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:41:14.544 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:14 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:14.541+0000 7fcea30a4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:41:14.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:14 smithi080 ceph-mon[27564]: Standby manager daemon z restarted 2024-03-04T15:41:14.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:14 smithi080 ceph-mon[27564]: Standby manager daemon z started 2024-03-04T15:41:14.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:14 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.87:0/3148068781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T15:41:14.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:14 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.87:0/3148068781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:41:14.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:14 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.87:0/3148068781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T15:41:14.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:14 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.87:0/3148068781' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:41:14.847 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:14 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:14.439+0000 7f152a5c3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:41:14.847 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:14 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: [04/Mar/2024:15:41:14] ENGINE Bus STARTING 2024-03-04T15:41:14.847 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:14 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: CherryPy Checker: 2024-03-04T15:41:14.847 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:14 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: The Application mounted at '' has an empty config. 2024-03-04T15:41:14.847 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:14 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: 2024-03-04T15:41:14.847 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:14 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: [04/Mar/2024:15:41:14] ENGINE Serving on http://:::9283 2024-03-04T15:41:14.847 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:41:14 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x[29254]: [04/Mar/2024:15:41:14] ENGINE Bus STARTED 2024-03-04T15:41:14.877 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:14 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:14.765+0000 7fcea30a4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:41:15.140 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:14 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:41:14] ENGINE Bus STARTING 2024-03-04T15:41:15.140 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: CherryPy Checker: 2024-03-04T15:41:15.141 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: The Application mounted at '' has an empty config. 2024-03-04T15:41:15.141 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:41:15.141 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:41:15] ENGINE Serving on http://:::9283 2024-03-04T15:41:15.141 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:41:15] ENGINE Bus STARTED 2024-03-04T15:41:15.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-04T15:41:15.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: Standby manager daemon x restarted 2024-03-04T15:41:15.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: Standby manager daemon x started 2024-03-04T15:41:15.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.80:0/1385389083' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:41:15.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.80:0/1385389083' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:41:15.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.80:0/1385389083' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:41:15.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.? 172.21.15.80:0/1385389083' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:41:15.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: Active manager daemon y restarted 2024-03-04T15:41:15.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: Activating manager daemon y 2024-03-04T15:41:15.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: osdmap e32: 3 total, 3 up, 3 in 2024-03-04T15:41:15.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: mgrmap e19: y(active, starting, since 0.00933568s), standbys: x, z 2024-03-04T15:41:15.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:41:15.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:41:15.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:41:15.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:41:15.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:41:15.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T15:41:15.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:41:15.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:41:15.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:41:15.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:41:15.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:41:15.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:41:15.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: Manager daemon y is now available 2024-03-04T15:41:15.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:15.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:41:15.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:41:15.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi087", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:41:15.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi080", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:41:15.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi087", "name": "osd_memory_target"}]': finished 2024-03-04T15:41:15.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi005", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:41:15.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi080", "name": "osd_memory_target"}]': finished 2024-03-04T15:41:15.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi005", "name": "osd_memory_target"}]': finished 2024-03-04T15:41:15.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:41:15.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:41:15.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:15.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:41:15.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:41:15.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:15 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:41:15.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-04T15:41:15.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: Standby manager daemon x restarted 2024-03-04T15:41:15.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: Standby manager daemon x started 2024-03-04T15:41:15.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.80:0/1385389083' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:41:15.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.80:0/1385389083' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:41:15.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.80:0/1385389083' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:41:15.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.? 172.21.15.80:0/1385389083' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:41:15.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: Active manager daemon y restarted 2024-03-04T15:41:15.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: Activating manager daemon y 2024-03-04T15:41:15.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: osdmap e32: 3 total, 3 up, 3 in 2024-03-04T15:41:15.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: mgrmap e19: y(active, starting, since 0.00933568s), standbys: x, z 2024-03-04T15:41:15.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:41:15.598 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:41:15.598 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:41:15.598 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:41:15.598 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:41:15.598 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T15:41:15.598 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:41:15.598 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:41:15.598 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:41:15.598 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:41:15.598 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:41:15.598 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:41:15.598 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: Manager daemon y is now available 2024-03-04T15:41:15.598 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:15.598 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:41:15.598 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:41:15.599 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi087", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:41:15.599 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi080", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:41:15.599 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi087", "name": "osd_memory_target"}]': finished 2024-03-04T15:41:15.599 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi005", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:41:15.599 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi080", "name": "osd_memory_target"}]': finished 2024-03-04T15:41:15.599 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi005", "name": "osd_memory_target"}]': finished 2024-03-04T15:41:15.599 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:41:15.599 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:41:15.599 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:15.599 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:41:15.599 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:41:15.599 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:15 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:41:15.646 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-04T15:41:15.646 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: Standby manager daemon x restarted 2024-03-04T15:41:15.646 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: Standby manager daemon x started 2024-03-04T15:41:15.646 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.80:0/1385389083' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:41:15.646 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.80:0/1385389083' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:41:15.647 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.80:0/1385389083' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:41:15.647 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.? 172.21.15.80:0/1385389083' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:41:15.647 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: Active manager daemon y restarted 2024-03-04T15:41:15.647 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: Activating manager daemon y 2024-03-04T15:41:15.647 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: osdmap e32: 3 total, 3 up, 3 in 2024-03-04T15:41:15.647 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: mgrmap e19: y(active, starting, since 0.00933568s), standbys: x, z 2024-03-04T15:41:15.647 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:41:15.647 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:41:15.647 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:41:15.647 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:41:15.647 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:41:15.647 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T15:41:15.647 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:41:15.647 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:41:15.648 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:41:15.648 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:41:15.648 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:41:15.648 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:41:15.648 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: Manager daemon y is now available 2024-03-04T15:41:15.648 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:15.648 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:41:15.648 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:41:15.648 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi087", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:41:15.648 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi080", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:41:15.648 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi087", "name": "osd_memory_target"}]': finished 2024-03-04T15:41:15.648 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi005", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:41:15.648 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi080", "name": "osd_memory_target"}]': finished 2024-03-04T15:41:15.648 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi005", "name": "osd_memory_target"}]': finished 2024-03-04T15:41:15.648 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:41:15.649 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:41:15.649 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:15.649 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:41:15.649 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:41:15.649 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:15 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:41:16.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-mon[25858]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-04T15:41:16.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-mon[25858]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-04T15:41:16.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-mon[25858]: Updating smithi087:/etc/ceph/ceph.conf 2024-03-04T15:41:16.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-mon[25858]: Updating smithi005:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:41:16.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-mon[25858]: Updating smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:41:16.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-mon[25858]: Updating smithi080:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:41:16.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-mon[25858]: mgrmap e20: y(active, since 1.0157s), standbys: x, z 2024-03-04T15:41:16.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:16 smithi087 ceph-mon[27664]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-04T15:41:16.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:16 smithi087 ceph-mon[27664]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-04T15:41:16.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:16 smithi087 ceph-mon[27664]: Updating smithi087:/etc/ceph/ceph.conf 2024-03-04T15:41:16.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:16 smithi087 ceph-mon[27664]: Updating smithi005:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:41:16.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:16 smithi087 ceph-mon[27664]: Updating smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:41:16.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:16 smithi087 ceph-mon[27664]: Updating smithi080:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:41:16.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:16 smithi087 ceph-mon[27664]: mgrmap e20: y(active, since 1.0157s), standbys: x, z 2024-03-04T15:41:16.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:16 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:16 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:16 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:16 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:16 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:16 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:16 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:16 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:16 smithi080 ceph-mon[27564]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-04T15:41:16.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:16 smithi080 ceph-mon[27564]: Updating smithi080:/etc/ceph/ceph.conf 2024-03-04T15:41:16.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:16 smithi080 ceph-mon[27564]: Updating smithi087:/etc/ceph/ceph.conf 2024-03-04T15:41:16.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:16 smithi080 ceph-mon[27564]: Updating smithi005:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:41:16.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:16 smithi080 ceph-mon[27564]: Updating smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:41:16.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:16 smithi080 ceph-mon[27564]: Updating smithi080:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:41:16.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:16 smithi080 ceph-mon[27564]: mgrmap e20: y(active, since 1.0157s), standbys: x, z 2024-03-04T15:41:16.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:16 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:16 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:16 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:16 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:16 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:16 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:16 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:16.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:16 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:17.145 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:16 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: ::ffff:172.21.15.80 - - [04/Mar/2024:15:41:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T15:41:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:17 smithi005 ceph-mon[25858]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:41:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:17 smithi005 ceph-mon[25858]: Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:41:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:17 smithi005 ceph-mon[25858]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:41:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:17 smithi005 ceph-mon[25858]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:17 smithi005 ceph-mon[25858]: Updating smithi005:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:41:17.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:17 smithi005 ceph-mon[25858]: Updating smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:41:17.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:17 smithi005 ceph-mon[25858]: Updating smithi080:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:41:17.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:17 smithi005 ceph-mon[25858]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:41:17.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:17 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:41:17.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:17 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:41:17.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:17 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:41:17.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:17 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:17.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:17 smithi005 ceph-mon[25858]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:41:17.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:17 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi005", "caps": []}]: dispatch 2024-03-04T15:41:17.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:17 smithi087 ceph-mon[27664]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:41:17.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:17 smithi087 ceph-mon[27664]: Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:41:17.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:17 smithi087 ceph-mon[27664]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:41:17.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:17 smithi087 ceph-mon[27664]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:17.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:17 smithi087 ceph-mon[27664]: Updating smithi005:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:41:17.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:17 smithi087 ceph-mon[27664]: Updating smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:41:17.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:17 smithi087 ceph-mon[27664]: Updating smithi080:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:41:17.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:17 smithi087 ceph-mon[27664]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:41:17.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:17 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:41:17.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:17 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:41:17.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:17 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:41:17.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:17 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:17.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:17 smithi087 ceph-mon[27664]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:41:17.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:17 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi005", "caps": []}]: dispatch 2024-03-04T15:41:17.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:17 smithi080 ceph-mon[27564]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:41:17.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:17 smithi080 ceph-mon[27564]: Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:41:17.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:17 smithi080 ceph-mon[27564]: Updating smithi080:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:41:17.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:17 smithi080 ceph-mon[27564]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:17.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:17 smithi080 ceph-mon[27564]: Updating smithi005:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:41:17.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:17 smithi080 ceph-mon[27564]: Updating smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:41:17.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:17 smithi080 ceph-mon[27564]: Updating smithi080:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:41:17.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:17 smithi080 ceph-mon[27564]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:41:17.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:17 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:41:17.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:17 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:41:17.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:17 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:41:17.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:17 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:17.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:17 smithi080 ceph-mon[27564]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:41:17.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:17 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi005", "caps": []}]: dispatch 2024-03-04T15:41:18.513 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:18 smithi005 ceph-mon[25858]: [04/Mar/2024:15:41:16] ENGINE Bus STARTING 2024-03-04T15:41:18.513 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:18 smithi005 ceph-mon[25858]: [04/Mar/2024:15:41:16] ENGINE Serving on http://172.21.15.5:8765 2024-03-04T15:41:18.513 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:18 smithi005 ceph-mon[25858]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:18.514 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:18 smithi005 ceph-mon[25858]: [04/Mar/2024:15:41:16] ENGINE Serving on https://172.21.15.5:7151 2024-03-04T15:41:18.514 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:18 smithi005 ceph-mon[25858]: [04/Mar/2024:15:41:16] ENGINE Bus STARTED 2024-03-04T15:41:18.514 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:18 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi080", "caps": []}]: dispatch 2024-03-04T15:41:18.514 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:18 smithi005 ceph-mon[25858]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-04T15:41:18.514 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:18 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi087", "caps": []}]: dispatch 2024-03-04T15:41:18.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:18 smithi087 ceph-mon[27664]: [04/Mar/2024:15:41:16] ENGINE Bus STARTING 2024-03-04T15:41:18.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:18 smithi087 ceph-mon[27664]: [04/Mar/2024:15:41:16] ENGINE Serving on http://172.21.15.5:8765 2024-03-04T15:41:18.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:18 smithi087 ceph-mon[27664]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:18.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:18 smithi087 ceph-mon[27664]: [04/Mar/2024:15:41:16] ENGINE Serving on https://172.21.15.5:7151 2024-03-04T15:41:18.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:18 smithi087 ceph-mon[27664]: [04/Mar/2024:15:41:16] ENGINE Bus STARTED 2024-03-04T15:41:18.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:18 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi080", "caps": []}]: dispatch 2024-03-04T15:41:18.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:18 smithi087 ceph-mon[27664]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-04T15:41:18.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:18 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi087", "caps": []}]: dispatch 2024-03-04T15:41:18.550 INFO:teuthology.orchestra.run.smithi005.stdout:77309411340 2024-03-04T15:41:18.550 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-04T15:41:18.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:18 smithi080 ceph-mon[27564]: [04/Mar/2024:15:41:16] ENGINE Bus STARTING 2024-03-04T15:41:18.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:18 smithi080 ceph-mon[27564]: [04/Mar/2024:15:41:16] ENGINE Serving on http://172.21.15.5:8765 2024-03-04T15:41:18.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:18 smithi080 ceph-mon[27564]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:18.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:18 smithi080 ceph-mon[27564]: [04/Mar/2024:15:41:16] ENGINE Serving on https://172.21.15.5:7151 2024-03-04T15:41:18.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:18 smithi080 ceph-mon[27564]: [04/Mar/2024:15:41:16] ENGINE Bus STARTED 2024-03-04T15:41:18.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:18 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi080", "caps": []}]: dispatch 2024-03-04T15:41:18.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:18 smithi080 ceph-mon[27564]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-04T15:41:18.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:18 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi087", "caps": []}]: dispatch 2024-03-04T15:41:19.115 INFO:teuthology.orchestra.run.smithi005.stdout:34359738391 2024-03-04T15:41:19.115 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-04T15:41:19.654 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:19 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:19 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:19 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:41:19.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:19 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi087", "caps": []}]: dispatch 2024-03-04T15:41:19.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:19 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi080", "caps": []}]: dispatch 2024-03-04T15:41:19.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:19 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:19 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:19 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:19 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.717 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:19.775 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:19 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.775 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:19 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.775 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:19 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:41:19.775 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:19 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi087", "caps": []}]: dispatch 2024-03-04T15:41:19.775 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:19 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi080", "caps": []}]: dispatch 2024-03-04T15:41:19.775 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:19 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.775 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:19 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.775 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:19 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.775 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:19 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:19 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:19 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:19 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:41:19.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:19 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi087", "caps": []}]: dispatch 2024-03-04T15:41:19.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:19 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi080", "caps": []}]: dispatch 2024-03-04T15:41:19.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:19 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:19 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:19 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:19.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:19 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:20.145 INFO:teuthology.orchestra.run.smithi005.stdout:55834574865 2024-03-04T15:41:20.146 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-04T15:41:20.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:20 smithi005 ceph-mon[25858]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:20.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:20 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:20.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:20 smithi005 ceph-mon[25858]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-04T15:41:20.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:20 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:20.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:20 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:20.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:20 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:20.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:20 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:20.713 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:20 smithi080 ceph-mon[27564]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:20.713 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:20 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:20.714 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:20 smithi080 ceph-mon[27564]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-04T15:41:20.714 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:20 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:20.714 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:20 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:20.714 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:20 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:20.714 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:20 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:20.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:20 smithi087 ceph-mon[27664]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:20.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:20 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:20.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:20 smithi087 ceph-mon[27664]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-04T15:41:20.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:20 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:20.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:20 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:20.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:20 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:20.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:20 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.075 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:21.726 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.726 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.726 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.726 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.726 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-04T15:41:21.726 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.726 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:41:21.726 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:41:21.727 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.727 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:41:21.727 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.727 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.727 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.727 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.727 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.727 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.727 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.727 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.727 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:21 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-04T15:41:21.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:41:21.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:41:21.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:41:21.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:21 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.921 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.922 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.922 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.922 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.922 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-04T15:41:21.922 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.922 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:41:21.922 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:41:21.922 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.922 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-04T15:41:21.922 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.922 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.922 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.922 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.923 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.923 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.923 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.923 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.923 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:21 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:21.942 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:22.825 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:22 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.825 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:22 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.825 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:22 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.825 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:22 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.825 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:22 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.825 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:22 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.825 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:22 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.825 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:22 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:22 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:22 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:22 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:22 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:22 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:22 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:22 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:22 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.975 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:22 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:22 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:22 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:22 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:22 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:22 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:22 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:22 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:22 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:22.976 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:22 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:23.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:22 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:23.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:22 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:23.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:22 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:23.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:22 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:23.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:22 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:23.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:22 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:23.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:22 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:23.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:22 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:23.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:22 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:23.097 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:22 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:23.098 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:22 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:23.098 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:22 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:23.098 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:22 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:23.812 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:23 smithi005 ceph-mon[25858]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-04T15:41:23.976 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:23 smithi080 ceph-mon[27564]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-04T15:41:23.980 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:23 smithi087 ceph-mon[27664]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-04T15:41:24.639 INFO:teuthology.orchestra.run.smithi005.stdout:77309411341 2024-03-04T15:41:24.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:24 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:24.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:24 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:24.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:24 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:24.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:24 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:41:24.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:24 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:41:24.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:24 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:41:24.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:24 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:24.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:24 smithi080 ceph-mon[27564]: Deploying daemon alertmanager.a on smithi005 2024-03-04T15:41:24.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:24 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/884376807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T15:41:24.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:24 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:24.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:24 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:24.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:24 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:24.864 INFO:teuthology.orchestra.run.smithi005.stdout:34359738392 2024-03-04T15:41:25.006 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:24 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:25.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:24 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:25.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:24 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:25.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:24 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:41:25.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:24 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:41:25.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:24 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:41:25.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:24 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:25.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:24 smithi087 ceph-mon[27664]: Deploying daemon alertmanager.a on smithi005 2024-03-04T15:41:25.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:24 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/884376807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T15:41:25.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:24 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:25.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:24 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:25.007 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:24 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:25.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:24 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:25.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:24 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:25.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:24 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:25.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:24 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:41:25.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:24 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:41:25.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:24 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:41:25.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:24 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:25.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:24 smithi005 ceph-mon[25858]: Deploying daemon alertmanager.a on smithi005 2024-03-04T15:41:25.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:24 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/884376807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T15:41:25.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:24 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:25.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:24 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:25.053 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:24 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:25.394 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411340 got 77309411341 for osd.2 2024-03-04T15:41:25.394 DEBUG:teuthology.parallel:result is None 2024-03-04T15:41:26.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:25 smithi087 ceph-mon[27664]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-04T15:41:26.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:25 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/4241685270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T15:41:26.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:25 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:26.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:25 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:26.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:25 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:26.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:25 smithi080 ceph-mon[27564]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-04T15:41:26.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:25 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/4241685270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T15:41:26.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:25 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:26.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:25 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:26.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:25 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:26.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:25 smithi005 ceph-mon[25858]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-04T15:41:26.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:25 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/4241685270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T15:41:26.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:25 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:26.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:25 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:26.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:25 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:27.110 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:26 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: ::ffff:172.21.15.80 - - [04/Mar/2024:15:41:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T15:41:27.135 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738391 got 34359738392 for osd.0 2024-03-04T15:41:27.135 DEBUG:teuthology.parallel:result is None 2024-03-04T15:41:27.864 INFO:teuthology.orchestra.run.smithi005.stdout:55834574867 2024-03-04T15:41:28.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:27 smithi005 ceph-mon[25858]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T15:41:28.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:27 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:28.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:27 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:28.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:27 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:28.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:27 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/2350447966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T15:41:28.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:27 smithi087 ceph-mon[27664]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T15:41:28.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:27 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:28.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:27 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:28.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:27 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:28.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:27 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/2350447966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T15:41:28.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:27 smithi080 ceph-mon[27564]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T15:41:28.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:27 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:28.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:27 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:28.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:27 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:28.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:27 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/2350447966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T15:41:29.290 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574865 got 55834574867 for osd.1 2024-03-04T15:41:29.290 DEBUG:teuthology.parallel:result is None 2024-03-04T15:41:29.290 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-04T15:41:29.291 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T15:41:29.972 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:30.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:29 smithi005 ceph-mon[25858]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-04T15:41:30.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:29 smithi087 ceph-mon[27664]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-04T15:41:30.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:29 smithi080 ceph-mon[27564]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-04T15:41:31.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:30 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:41:31.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:30 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:41:31.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:30 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:41:32.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:31 smithi005 ceph-mon[25858]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-04T15:41:32.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:31 smithi087 ceph-mon[27664]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-04T15:41:32.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:31 smithi080 ceph-mon[27564]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-04T15:41:34.054 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:41:34.057 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-03-04T15:41:34.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:33 smithi005 ceph-mon[25858]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:34.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:33 smithi087 ceph-mon[27664]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:34.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:33 smithi080 ceph-mon[27564]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:35.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:34 smithi005 ceph-mon[25858]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:41:35.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:34 smithi087 ceph-mon[27664]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:41:35.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:34 smithi080 ceph-mon[27564]: from='client.14514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:41:35.928 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-03-04T15:41:32.797794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96636,"kb_used_data":3108,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126532,"statfs":{"total":287972524032,"available":287873568768,"internally_reserved":0,"allocated":3182592,"data_stored":2109504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003046"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785025+0000","last_change":"2024-03-04T15:40:51.769959+0000","last_active":"2024-03-04T15:41:14.785025+0000","last_peered":"2024-03-04T15:41:14.785025+0000","last_clean":"2024-03-04T15:41:14.785025+0000","last_became_active":"2024-03-04T15:40:51.769564+0000","last_became_peered":"2024-03-04T15:40:51.769564+0000","last_unstale":"2024-03-04T15:41:14.785025+0000","last_undegraded":"2024-03-04T15:41:14.785025+0000","last_fullsized":"2024-03-04T15:41:14.785025+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:13:03.634445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788256+0000","last_change":"2024-03-04T15:40:49.757616+0000","last_active":"2024-03-04T15:41:14.788256+0000","last_peered":"2024-03-04T15:41:14.788256+0000","last_clean":"2024-03-04T15:41:14.788256+0000","last_became_active":"2024-03-04T15:40:49.757327+0000","last_became_peered":"2024-03-04T15:40:49.757327+0000","last_unstale":"2024-03-04T15:41:14.788256+0000","last_undegraded":"2024-03-04T15:41:14.788256+0000","last_fullsized":"2024-03-04T15:41:14.788256+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:00:53.575547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786670+0000","last_change":"2024-03-04T15:40:45.735989+0000","last_active":"2024-03-04T15:41:14.786670+0000","last_peered":"2024-03-04T15:41:14.786670+0000","last_clean":"2024-03-04T15:41:14.786670+0000","last_became_active":"2024-03-04T15:40:45.735018+0000","last_became_peered":"2024-03-04T15:40:45.735018+0000","last_unstale":"2024-03-04T15:41:14.786670+0000","last_undegraded":"2024-03-04T15:41:14.786670+0000","last_fullsized":"2024-03-04T15:41:14.786670+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:26:56.697354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786032+0000","last_change":"2024-03-04T15:40:47.740686+0000","last_active":"2024-03-04T15:41:14.786032+0000","last_peered":"2024-03-04T15:41:14.786032+0000","last_clean":"2024-03-04T15:41:14.786032+0000","last_became_active":"2024-03-04T15:40:47.740005+0000","last_became_peered":"2024-03-04T15:40:47.740005+0000","last_unstale":"2024-03-04T15:41:14.786032+0000","last_undegraded":"2024-03-04T15:41:14.786032+0000","last_fullsized":"2024-03-04T15:41:14.786032+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:13:06.487583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783462+0000","last_change":"2024-03-04T15:40:51.765009+0000","last_active":"2024-03-04T15:41:14.783462+0000","last_peered":"2024-03-04T15:41:14.783462+0000","last_clean":"2024-03-04T15:41:14.783462+0000","last_became_active":"2024-03-04T15:40:51.764154+0000","last_became_peered":"2024-03-04T15:40:51.764154+0000","last_unstale":"2024-03-04T15:41:14.783462+0000","last_undegraded":"2024-03-04T15:41:14.783462+0000","last_fullsized":"2024-03-04T15:41:14.783462+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:24:32.866079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784748+0000","last_change":"2024-03-04T15:40:49.756733+0000","last_active":"2024-03-04T15:41:14.784748+0000","last_peered":"2024-03-04T15:41:14.784748+0000","last_clean":"2024-03-04T15:41:14.784748+0000","last_became_active":"2024-03-04T15:40:49.756337+0000","last_became_peered":"2024-03-04T15:40:49.756337+0000","last_unstale":"2024-03-04T15:41:14.784748+0000","last_undegraded":"2024-03-04T15:41:14.784748+0000","last_fullsized":"2024-03-04T15:41:14.784748+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:47:18.124798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785685+0000","last_change":"2024-03-04T15:40:45.735614+0000","last_active":"2024-03-04T15:41:14.785685+0000","last_peered":"2024-03-04T15:41:14.785685+0000","last_clean":"2024-03-04T15:41:14.785685+0000","last_became_active":"2024-03-04T15:40:45.734940+0000","last_became_peered":"2024-03-04T15:40:45.734940+0000","last_unstale":"2024-03-04T15:41:14.785685+0000","last_undegraded":"2024-03-04T15:41:14.785685+0000","last_fullsized":"2024-03-04T15:41:14.785685+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:45:36.911780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786502+0000","last_change":"2024-03-04T15:40:47.740459+0000","last_active":"2024-03-04T15:41:14.786502+0000","last_peered":"2024-03-04T15:41:14.786502+0000","last_clean":"2024-03-04T15:41:14.786502+0000","last_became_active":"2024-03-04T15:40:47.739339+0000","last_became_peered":"2024-03-04T15:40:47.739339+0000","last_unstale":"2024-03-04T15:41:14.786502+0000","last_undegraded":"2024-03-04T15:41:14.786502+0000","last_fullsized":"2024-03-04T15:41:14.786502+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:48:25.179538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788275+0000","last_change":"2024-03-04T15:40:51.767954+0000","last_active":"2024-03-04T15:41:14.788275+0000","last_peered":"2024-03-04T15:41:14.788275+0000","last_clean":"2024-03-04T15:41:14.788275+0000","last_became_active":"2024-03-04T15:40:51.767645+0000","last_became_peered":"2024-03-04T15:40:51.767645+0000","last_unstale":"2024-03-04T15:41:14.788275+0000","last_undegraded":"2024-03-04T15:41:14.788275+0000","last_fullsized":"2024-03-04T15:41:14.788275+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:32:46.739642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784899+0000","last_change":"2024-03-04T15:40:49.757496+0000","last_active":"2024-03-04T15:41:14.784899+0000","last_peered":"2024-03-04T15:41:14.784899+0000","last_clean":"2024-03-04T15:41:14.784899+0000","last_became_active":"2024-03-04T15:40:49.757186+0000","last_became_peered":"2024-03-04T15:40:49.757186+0000","last_unstale":"2024-03-04T15:41:14.784899+0000","last_undegraded":"2024-03-04T15:41:14.784899+0000","last_fullsized":"2024-03-04T15:41:14.784899+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:12:38.250653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.791351+0000","last_change":"2024-03-04T15:40:45.735621+0000","last_active":"2024-03-04T15:41:14.791351+0000","last_peered":"2024-03-04T15:41:14.791351+0000","last_clean":"2024-03-04T15:41:14.791351+0000","last_became_active":"2024-03-04T15:40:45.734748+0000","last_became_peered":"2024-03-04T15:40:45.734748+0000","last_unstale":"2024-03-04T15:41:14.791351+0000","last_undegraded":"2024-03-04T15:41:14.791351+0000","last_fullsized":"2024-03-04T15:41:14.791351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:11.985815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786207+0000","last_change":"2024-03-04T15:40:47.743651+0000","last_active":"2024-03-04T15:41:14.786207+0000","last_peered":"2024-03-04T15:41:14.786207+0000","last_clean":"2024-03-04T15:41:14.786207+0000","last_became_active":"2024-03-04T15:40:47.743147+0000","last_became_peered":"2024-03-04T15:40:47.743147+0000","last_unstale":"2024-03-04T15:41:14.786207+0000","last_undegraded":"2024-03-04T15:41:14.786207+0000","last_fullsized":"2024-03-04T15:41:14.786207+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:21.900919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784721+0000","last_change":"2024-03-04T15:40:51.764182+0000","last_active":"2024-03-04T15:41:14.784721+0000","last_peered":"2024-03-04T15:41:14.784721+0000","last_clean":"2024-03-04T15:41:14.784721+0000","last_became_active":"2024-03-04T15:40:51.762173+0000","last_became_peered":"2024-03-04T15:40:51.762173+0000","last_unstale":"2024-03-04T15:41:14.784721+0000","last_undegraded":"2024-03-04T15:41:14.784721+0000","last_fullsized":"2024-03-04T15:41:14.784721+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:38:07.307651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784535+0000","last_change":"2024-03-04T15:40:49.756262+0000","last_active":"2024-03-04T15:41:14.784535+0000","last_peered":"2024-03-04T15:41:14.784535+0000","last_clean":"2024-03-04T15:41:14.784535+0000","last_became_active":"2024-03-04T15:40:49.753202+0000","last_became_peered":"2024-03-04T15:40:49.753202+0000","last_unstale":"2024-03-04T15:41:14.784535+0000","last_undegraded":"2024-03-04T15:41:14.784535+0000","last_fullsized":"2024-03-04T15:41:14.784535+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:32:41.115965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789700+0000","last_change":"2024-03-04T15:40:45.736529+0000","last_active":"2024-03-04T15:41:14.789700+0000","last_peered":"2024-03-04T15:41:14.789700+0000","last_clean":"2024-03-04T15:41:14.789700+0000","last_became_active":"2024-03-04T15:40:45.735582+0000","last_became_peered":"2024-03-04T15:40:45.735582+0000","last_unstale":"2024-03-04T15:41:14.789700+0000","last_undegraded":"2024-03-04T15:41:14.789700+0000","last_fullsized":"2024-03-04T15:41:14.789700+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:11:33.370775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.791253+0000","last_change":"2024-03-04T15:40:47.745175+0000","last_active":"2024-03-04T15:41:14.791253+0000","last_peered":"2024-03-04T15:41:14.791253+0000","last_clean":"2024-03-04T15:41:14.791253+0000","last_became_active":"2024-03-04T15:40:47.744868+0000","last_became_peered":"2024-03-04T15:40:47.744868+0000","last_unstale":"2024-03-04T15:41:14.791253+0000","last_undegraded":"2024-03-04T15:41:14.791253+0000","last_fullsized":"2024-03-04T15:41:14.791253+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:30:16.660951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785792+0000","last_change":"2024-03-04T15:40:51.762535+0000","last_active":"2024-03-04T15:41:14.785792+0000","last_peered":"2024-03-04T15:41:14.785792+0000","last_clean":"2024-03-04T15:41:14.785792+0000","last_became_active":"2024-03-04T15:40:51.762198+0000","last_became_peered":"2024-03-04T15:40:51.762198+0000","last_unstale":"2024-03-04T15:41:14.785792+0000","last_undegraded":"2024-03-04T15:41:14.785792+0000","last_fullsized":"2024-03-04T15:41:14.785792+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:33:53.857497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.790706+0000","last_change":"2024-03-04T15:40:49.758011+0000","last_active":"2024-03-04T15:41:14.790706+0000","last_peered":"2024-03-04T15:41:14.790706+0000","last_clean":"2024-03-04T15:41:14.790706+0000","last_became_active":"2024-03-04T15:40:49.757654+0000","last_became_peered":"2024-03-04T15:40:49.757654+0000","last_unstale":"2024-03-04T15:41:14.790706+0000","last_undegraded":"2024-03-04T15:41:14.790706+0000","last_fullsized":"2024-03-04T15:41:14.790706+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:20:37.542151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789211+0000","last_change":"2024-03-04T15:40:45.736433+0000","last_active":"2024-03-04T15:41:14.789211+0000","last_peered":"2024-03-04T15:41:14.789211+0000","last_clean":"2024-03-04T15:41:14.789211+0000","last_became_active":"2024-03-04T15:40:45.735672+0000","last_became_peered":"2024-03-04T15:40:45.735672+0000","last_unstale":"2024-03-04T15:41:14.789211+0000","last_undegraded":"2024-03-04T15:41:14.789211+0000","last_fullsized":"2024-03-04T15:41:14.789211+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:33:12.025991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789261+0000","last_change":"2024-03-04T15:40:47.741747+0000","last_active":"2024-03-04T15:41:14.789261+0000","last_peered":"2024-03-04T15:41:14.789261+0000","last_clean":"2024-03-04T15:41:14.789261+0000","last_became_active":"2024-03-04T15:40:47.741435+0000","last_became_peered":"2024-03-04T15:40:47.741435+0000","last_unstale":"2024-03-04T15:41:14.789261+0000","last_undegraded":"2024-03-04T15:41:14.789261+0000","last_fullsized":"2024-03-04T15:41:14.789261+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:52:19.519700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785676+0000","last_change":"2024-03-04T15:40:51.769036+0000","last_active":"2024-03-04T15:41:14.785676+0000","last_peered":"2024-03-04T15:41:14.785676+0000","last_clean":"2024-03-04T15:41:14.785676+0000","last_became_active":"2024-03-04T15:40:51.768615+0000","last_became_peered":"2024-03-04T15:40:51.768615+0000","last_unstale":"2024-03-04T15:41:14.785676+0000","last_undegraded":"2024-03-04T15:41:14.785676+0000","last_fullsized":"2024-03-04T15:41:14.785676+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:01:05.970415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789824+0000","last_change":"2024-03-04T15:40:49.752517+0000","last_active":"2024-03-04T15:41:14.789824+0000","last_peered":"2024-03-04T15:41:14.789824+0000","last_clean":"2024-03-04T15:41:14.789824+0000","last_became_active":"2024-03-04T15:40:49.752161+0000","last_became_peered":"2024-03-04T15:40:49.752161+0000","last_unstale":"2024-03-04T15:41:14.789824+0000","last_undegraded":"2024-03-04T15:41:14.789824+0000","last_fullsized":"2024-03-04T15:41:14.789824+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:00:42.835911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785723+0000","last_change":"2024-03-04T15:40:45.734647+0000","last_active":"2024-03-04T15:41:14.785723+0000","last_peered":"2024-03-04T15:41:14.785723+0000","last_clean":"2024-03-04T15:41:14.785723+0000","last_became_active":"2024-03-04T15:40:45.733972+0000","last_became_peered":"2024-03-04T15:40:45.733972+0000","last_unstale":"2024-03-04T15:41:14.785723+0000","last_undegraded":"2024-03-04T15:41:14.785723+0000","last_fullsized":"2024-03-04T15:41:14.785723+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:21:37.130794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788933+0000","last_change":"2024-03-04T15:40:47.741957+0000","last_active":"2024-03-04T15:41:14.788933+0000","last_peered":"2024-03-04T15:41:14.788933+0000","last_clean":"2024-03-04T15:41:14.788933+0000","last_became_active":"2024-03-04T15:40:47.741458+0000","last_became_peered":"2024-03-04T15:40:47.741458+0000","last_unstale":"2024-03-04T15:41:14.788933+0000","last_undegraded":"2024-03-04T15:41:14.788933+0000","last_fullsized":"2024-03-04T15:41:14.788933+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:32:26.007477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784898+0000","last_change":"2024-03-04T15:40:51.769872+0000","last_active":"2024-03-04T15:41:14.784898+0000","last_peered":"2024-03-04T15:41:14.784898+0000","last_clean":"2024-03-04T15:41:14.784898+0000","last_became_active":"2024-03-04T15:40:51.769389+0000","last_became_peered":"2024-03-04T15:40:51.769389+0000","last_unstale":"2024-03-04T15:41:14.784898+0000","last_undegraded":"2024-03-04T15:41:14.784898+0000","last_fullsized":"2024-03-04T15:41:14.784898+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:46:49.503639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'8","reported_seq":43,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:25.782794+0000","last_change":"2024-03-04T15:40:49.758544+0000","last_active":"2024-03-04T15:41:25.782794+0000","last_peered":"2024-03-04T15:41:25.782794+0000","last_clean":"2024-03-04T15:41:25.782794+0000","last_became_active":"2024-03-04T15:40:49.757583+0000","last_became_peered":"2024-03-04T15:40:49.757583+0000","last_unstale":"2024-03-04T15:41:25.782794+0000","last_undegraded":"2024-03-04T15:41:25.782794+0000","last_fullsized":"2024-03-04T15:41:25.782794+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:15:49.658483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783886+0000","last_change":"2024-03-04T15:40:45.736001+0000","last_active":"2024-03-04T15:41:14.783886+0000","last_peered":"2024-03-04T15:41:14.783886+0000","last_clean":"2024-03-04T15:41:14.783886+0000","last_became_active":"2024-03-04T15:40:45.735635+0000","last_became_peered":"2024-03-04T15:40:45.735635+0000","last_unstale":"2024-03-04T15:41:14.783886+0000","last_undegraded":"2024-03-04T15:41:14.783886+0000","last_fullsized":"2024-03-04T15:41:14.783886+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:48:24.241284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783342+0000","last_change":"2024-03-04T15:40:47.741660+0000","last_active":"2024-03-04T15:41:14.783342+0000","last_peered":"2024-03-04T15:41:14.783342+0000","last_clean":"2024-03-04T15:41:14.783342+0000","last_became_active":"2024-03-04T15:40:47.740867+0000","last_became_peered":"2024-03-04T15:40:47.740867+0000","last_unstale":"2024-03-04T15:41:14.783342+0000","last_undegraded":"2024-03-04T15:41:14.783342+0000","last_fullsized":"2024-03-04T15:41:14.783342+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:43:52.782567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786067+0000","last_change":"2024-03-04T15:40:51.766417+0000","last_active":"2024-03-04T15:41:14.786067+0000","last_peered":"2024-03-04T15:41:14.786067+0000","last_clean":"2024-03-04T15:41:14.786067+0000","last_became_active":"2024-03-04T15:40:51.764751+0000","last_became_peered":"2024-03-04T15:40:51.764751+0000","last_unstale":"2024-03-04T15:41:14.786067+0000","last_undegraded":"2024-03-04T15:41:14.786067+0000","last_fullsized":"2024-03-04T15:41:14.786067+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:21:11.249731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785201+0000","last_change":"2024-03-04T15:40:49.757575+0000","last_active":"2024-03-04T15:41:14.785201+0000","last_peered":"2024-03-04T15:41:14.785201+0000","last_clean":"2024-03-04T15:41:14.785201+0000","last_became_active":"2024-03-04T15:40:49.757172+0000","last_became_peered":"2024-03-04T15:40:49.757172+0000","last_unstale":"2024-03-04T15:41:14.785201+0000","last_undegraded":"2024-03-04T15:41:14.785201+0000","last_fullsized":"2024-03-04T15:41:14.785201+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:29.152753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788299+0000","last_change":"2024-03-04T15:40:45.737162+0000","last_active":"2024-03-04T15:41:14.788299+0000","last_peered":"2024-03-04T15:41:14.788299+0000","last_clean":"2024-03-04T15:41:14.788299+0000","last_became_active":"2024-03-04T15:40:45.736401+0000","last_became_peered":"2024-03-04T15:40:45.736401+0000","last_unstale":"2024-03-04T15:41:14.788299+0000","last_undegraded":"2024-03-04T15:41:14.788299+0000","last_fullsized":"2024-03-04T15:41:14.788299+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:27:34.287422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783430+0000","last_change":"2024-03-04T15:40:47.741535+0000","last_active":"2024-03-04T15:41:14.783430+0000","last_peered":"2024-03-04T15:41:14.783430+0000","last_clean":"2024-03-04T15:41:14.783430+0000","last_became_active":"2024-03-04T15:40:47.741016+0000","last_became_peered":"2024-03-04T15:40:47.741016+0000","last_unstale":"2024-03-04T15:41:14.783430+0000","last_undegraded":"2024-03-04T15:41:14.783430+0000","last_fullsized":"2024-03-04T15:41:14.783430+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:23:53.943756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784954+0000","last_change":"2024-03-04T15:40:51.769971+0000","last_active":"2024-03-04T15:41:14.784954+0000","last_peered":"2024-03-04T15:41:14.784954+0000","last_clean":"2024-03-04T15:41:14.784954+0000","last_became_active":"2024-03-04T15:40:51.769628+0000","last_became_peered":"2024-03-04T15:40:51.769628+0000","last_unstale":"2024-03-04T15:41:14.784954+0000","last_undegraded":"2024-03-04T15:41:14.784954+0000","last_fullsized":"2024-03-04T15:41:14.784954+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:47:50.538893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783345+0000","last_change":"2024-03-04T15:40:49.760118+0000","last_active":"2024-03-04T15:41:14.783345+0000","last_peered":"2024-03-04T15:41:14.783345+0000","last_clean":"2024-03-04T15:41:14.783345+0000","last_became_active":"2024-03-04T15:40:49.759629+0000","last_became_peered":"2024-03-04T15:40:49.759629+0000","last_unstale":"2024-03-04T15:41:14.783345+0000","last_undegraded":"2024-03-04T15:41:14.783345+0000","last_fullsized":"2024-03-04T15:41:14.783345+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:53:49.102411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787089+0000","last_change":"2024-03-04T15:40:45.739273+0000","last_active":"2024-03-04T15:41:14.787089+0000","last_peered":"2024-03-04T15:41:14.787089+0000","last_clean":"2024-03-04T15:41:14.787089+0000","last_became_active":"2024-03-04T15:40:45.738854+0000","last_became_peered":"2024-03-04T15:40:45.738854+0000","last_unstale":"2024-03-04T15:41:14.787089+0000","last_undegraded":"2024-03-04T15:41:14.787089+0000","last_fullsized":"2024-03-04T15:41:14.787089+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:57:03.191782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786057+0000","last_change":"2024-03-04T15:40:47.742402+0000","last_active":"2024-03-04T15:41:14.786057+0000","last_peered":"2024-03-04T15:41:14.786057+0000","last_clean":"2024-03-04T15:41:14.786057+0000","last_became_active":"2024-03-04T15:40:47.741747+0000","last_became_peered":"2024-03-04T15:40:47.741747+0000","last_unstale":"2024-03-04T15:41:14.786057+0000","last_undegraded":"2024-03-04T15:41:14.786057+0000","last_fullsized":"2024-03-04T15:41:14.786057+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:53:37.578353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785013+0000","last_change":"2024-03-04T15:40:51.762576+0000","last_active":"2024-03-04T15:41:14.785013+0000","last_peered":"2024-03-04T15:41:14.785013+0000","last_clean":"2024-03-04T15:41:14.785013+0000","last_became_active":"2024-03-04T15:40:51.762104+0000","last_became_peered":"2024-03-04T15:40:51.762104+0000","last_unstale":"2024-03-04T15:41:14.785013+0000","last_undegraded":"2024-03-04T15:41:14.785013+0000","last_fullsized":"2024-03-04T15:41:14.785013+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:47:33.290012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783647+0000","last_change":"2024-03-04T15:40:49.759706+0000","last_active":"2024-03-04T15:41:14.783647+0000","last_peered":"2024-03-04T15:41:14.783647+0000","last_clean":"2024-03-04T15:41:14.783647+0000","last_became_active":"2024-03-04T15:40:49.758356+0000","last_became_peered":"2024-03-04T15:40:49.758356+0000","last_unstale":"2024-03-04T15:41:14.783647+0000","last_undegraded":"2024-03-04T15:41:14.783647+0000","last_fullsized":"2024-03-04T15:41:14.783647+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:19:29.972687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786154+0000","last_change":"2024-03-04T15:40:45.737848+0000","last_active":"2024-03-04T15:41:14.786154+0000","last_peered":"2024-03-04T15:41:14.786154+0000","last_clean":"2024-03-04T15:41:14.786154+0000","last_became_active":"2024-03-04T15:40:45.737534+0000","last_became_peered":"2024-03-04T15:40:45.737534+0000","last_unstale":"2024-03-04T15:41:14.786154+0000","last_undegraded":"2024-03-04T15:41:14.786154+0000","last_fullsized":"2024-03-04T15:41:14.786154+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:03:46.278852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"31'6","reported_seq":32,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786899+0000","last_change":"2024-03-04T15:40:47.741220+0000","last_active":"2024-03-04T15:41:14.786899+0000","last_peered":"2024-03-04T15:41:14.786899+0000","last_clean":"2024-03-04T15:41:14.786899+0000","last_became_active":"2024-03-04T15:40:47.740500+0000","last_became_peered":"2024-03-04T15:40:47.740500+0000","last_unstale":"2024-03-04T15:41:14.786899+0000","last_undegraded":"2024-03-04T15:41:14.786899+0000","last_fullsized":"2024-03-04T15:41:14.786899+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:10:21.366771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788442+0000","last_change":"2024-03-04T15:40:51.769503+0000","last_active":"2024-03-04T15:41:14.788442+0000","last_peered":"2024-03-04T15:41:14.788442+0000","last_clean":"2024-03-04T15:41:14.788442+0000","last_became_active":"2024-03-04T15:40:51.768444+0000","last_became_peered":"2024-03-04T15:40:51.768444+0000","last_unstale":"2024-03-04T15:41:14.788442+0000","last_undegraded":"2024-03-04T15:41:14.788442+0000","last_fullsized":"2024-03-04T15:41:14.788442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:57:56.690862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785080+0000","last_change":"2024-03-04T15:40:49.756625+0000","last_active":"2024-03-04T15:41:14.785080+0000","last_peered":"2024-03-04T15:41:14.785080+0000","last_clean":"2024-03-04T15:41:14.785080+0000","last_became_active":"2024-03-04T15:40:49.756281+0000","last_became_peered":"2024-03-04T15:40:49.756281+0000","last_unstale":"2024-03-04T15:41:14.785080+0000","last_undegraded":"2024-03-04T15:41:14.785080+0000","last_fullsized":"2024-03-04T15:41:14.785080+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:24:52.549448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.791180+0000","last_change":"2024-03-04T15:40:45.733948+0000","last_active":"2024-03-04T15:41:14.791180+0000","last_peered":"2024-03-04T15:41:14.791180+0000","last_clean":"2024-03-04T15:41:14.791180+0000","last_became_active":"2024-03-04T15:40:45.733532+0000","last_became_peered":"2024-03-04T15:40:45.733532+0000","last_unstale":"2024-03-04T15:41:14.791180+0000","last_undegraded":"2024-03-04T15:41:14.791180+0000","last_fullsized":"2024-03-04T15:41:14.791180+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:04:15.869340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787076+0000","last_change":"2024-03-04T15:40:47.743159+0000","last_active":"2024-03-04T15:41:14.787076+0000","last_peered":"2024-03-04T15:41:14.787076+0000","last_clean":"2024-03-04T15:41:14.787076+0000","last_became_active":"2024-03-04T15:40:47.742399+0000","last_became_peered":"2024-03-04T15:40:47.742399+0000","last_unstale":"2024-03-04T15:41:14.787076+0000","last_undegraded":"2024-03-04T15:41:14.787076+0000","last_fullsized":"2024-03-04T15:41:14.787076+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:35:27.075374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788432+0000","last_change":"2024-03-04T15:40:51.769547+0000","last_active":"2024-03-04T15:41:14.788432+0000","last_peered":"2024-03-04T15:41:14.788432+0000","last_clean":"2024-03-04T15:41:14.788432+0000","last_became_active":"2024-03-04T15:40:51.768527+0000","last_became_peered":"2024-03-04T15:40:51.768527+0000","last_unstale":"2024-03-04T15:41:14.788432+0000","last_undegraded":"2024-03-04T15:41:14.788432+0000","last_fullsized":"2024-03-04T15:41:14.788432+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:24:14.841245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784041+0000","last_change":"2024-03-04T15:40:49.760421+0000","last_active":"2024-03-04T15:41:14.784041+0000","last_peered":"2024-03-04T15:41:14.784041+0000","last_clean":"2024-03-04T15:41:14.784041+0000","last_became_active":"2024-03-04T15:40:49.760045+0000","last_became_peered":"2024-03-04T15:40:49.760045+0000","last_unstale":"2024-03-04T15:41:14.784041+0000","last_undegraded":"2024-03-04T15:41:14.784041+0000","last_fullsized":"2024-03-04T15:41:14.784041+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:15:52.398290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787209+0000","last_change":"2024-03-04T15:40:45.739369+0000","last_active":"2024-03-04T15:41:14.787209+0000","last_peered":"2024-03-04T15:41:14.787209+0000","last_clean":"2024-03-04T15:41:14.787209+0000","last_became_active":"2024-03-04T15:40:45.739017+0000","last_became_peered":"2024-03-04T15:40:45.739017+0000","last_unstale":"2024-03-04T15:41:14.787209+0000","last_undegraded":"2024-03-04T15:41:14.787209+0000","last_fullsized":"2024-03-04T15:41:14.787209+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:31:48.754552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.791125+0000","last_change":"2024-03-04T15:40:47.742047+0000","last_active":"2024-03-04T15:41:14.791125+0000","last_peered":"2024-03-04T15:41:14.791125+0000","last_clean":"2024-03-04T15:41:14.791125+0000","last_became_active":"2024-03-04T15:40:47.741599+0000","last_became_peered":"2024-03-04T15:40:47.741599+0000","last_unstale":"2024-03-04T15:41:14.791125+0000","last_undegraded":"2024-03-04T15:41:14.791125+0000","last_fullsized":"2024-03-04T15:41:14.791125+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:20:49.033294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786985+0000","last_change":"2024-03-04T15:40:51.770171+0000","last_active":"2024-03-04T15:41:14.786985+0000","last_peered":"2024-03-04T15:41:14.786985+0000","last_clean":"2024-03-04T15:41:14.786985+0000","last_became_active":"2024-03-04T15:40:51.769837+0000","last_became_peered":"2024-03-04T15:40:51.769837+0000","last_unstale":"2024-03-04T15:41:14.786985+0000","last_undegraded":"2024-03-04T15:41:14.786985+0000","last_fullsized":"2024-03-04T15:41:14.786985+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:46:32.095191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:25.782802+0000","last_change":"2024-03-04T15:40:49.750132+0000","last_active":"2024-03-04T15:41:25.782802+0000","last_peered":"2024-03-04T15:41:25.782802+0000","last_clean":"2024-03-04T15:41:25.782802+0000","last_became_active":"2024-03-04T15:40:49.749836+0000","last_became_peered":"2024-03-04T15:40:49.749836+0000","last_unstale":"2024-03-04T15:41:25.782802+0000","last_undegraded":"2024-03-04T15:41:25.782802+0000","last_fullsized":"2024-03-04T15:41:25.782802+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:48:59.571636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788811+0000","last_change":"2024-03-04T15:40:45.734935+0000","last_active":"2024-03-04T15:41:14.788811+0000","last_peered":"2024-03-04T15:41:14.788811+0000","last_clean":"2024-03-04T15:41:14.788811+0000","last_became_active":"2024-03-04T15:40:45.734080+0000","last_became_peered":"2024-03-04T15:40:45.734080+0000","last_unstale":"2024-03-04T15:41:14.788811+0000","last_undegraded":"2024-03-04T15:41:14.788811+0000","last_fullsized":"2024-03-04T15:41:14.788811+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:41:10.393533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784202+0000","last_change":"2024-03-04T15:40:47.739311+0000","last_active":"2024-03-04T15:41:14.784202+0000","last_peered":"2024-03-04T15:41:14.784202+0000","last_clean":"2024-03-04T15:41:14.784202+0000","last_became_active":"2024-03-04T15:40:47.738977+0000","last_became_peered":"2024-03-04T15:40:47.738977+0000","last_unstale":"2024-03-04T15:41:14.784202+0000","last_undegraded":"2024-03-04T15:41:14.784202+0000","last_fullsized":"2024-03-04T15:41:14.784202+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:00:06.410395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785257+0000","last_change":"2024-03-04T15:40:51.763798+0000","last_active":"2024-03-04T15:41:14.785257+0000","last_peered":"2024-03-04T15:41:14.785257+0000","last_clean":"2024-03-04T15:41:14.785257+0000","last_became_active":"2024-03-04T15:40:51.762400+0000","last_became_peered":"2024-03-04T15:40:51.762400+0000","last_unstale":"2024-03-04T15:41:14.785257+0000","last_undegraded":"2024-03-04T15:41:14.785257+0000","last_fullsized":"2024-03-04T15:41:14.785257+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:55:10.737732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:25.782831+0000","last_change":"2024-03-04T15:40:49.757829+0000","last_active":"2024-03-04T15:41:25.782831+0000","last_peered":"2024-03-04T15:41:25.782831+0000","last_clean":"2024-03-04T15:41:25.782831+0000","last_became_active":"2024-03-04T15:40:49.757491+0000","last_became_peered":"2024-03-04T15:40:49.757491+0000","last_unstale":"2024-03-04T15:41:25.782831+0000","last_undegraded":"2024-03-04T15:41:25.782831+0000","last_fullsized":"2024-03-04T15:41:25.782831+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:23:17.843177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788899+0000","last_change":"2024-03-04T15:40:45.733121+0000","last_active":"2024-03-04T15:41:14.788899+0000","last_peered":"2024-03-04T15:41:14.788899+0000","last_clean":"2024-03-04T15:41:14.788899+0000","last_became_active":"2024-03-04T15:40:45.732811+0000","last_became_peered":"2024-03-04T15:40:45.732811+0000","last_unstale":"2024-03-04T15:41:14.788899+0000","last_undegraded":"2024-03-04T15:41:14.788899+0000","last_fullsized":"2024-03-04T15:41:14.788899+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:57.484451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785455+0000","last_change":"2024-03-04T15:40:47.742600+0000","last_active":"2024-03-04T15:41:14.785455+0000","last_peered":"2024-03-04T15:41:14.785455+0000","last_clean":"2024-03-04T15:41:14.785455+0000","last_became_active":"2024-03-04T15:40:47.741503+0000","last_became_peered":"2024-03-04T15:40:47.741503+0000","last_unstale":"2024-03-04T15:41:14.785455+0000","last_undegraded":"2024-03-04T15:41:14.785455+0000","last_fullsized":"2024-03-04T15:41:14.785455+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:06:12.243951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785721+0000","last_change":"2024-03-04T15:40:51.764833+0000","last_active":"2024-03-04T15:41:14.785721+0000","last_peered":"2024-03-04T15:41:14.785721+0000","last_clean":"2024-03-04T15:41:14.785721+0000","last_became_active":"2024-03-04T15:40:51.764487+0000","last_became_peered":"2024-03-04T15:40:51.764487+0000","last_unstale":"2024-03-04T15:41:14.785721+0000","last_undegraded":"2024-03-04T15:41:14.785721+0000","last_fullsized":"2024-03-04T15:41:14.785721+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:45:43.949495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786340+0000","last_change":"2024-03-04T15:40:49.760632+0000","last_active":"2024-03-04T15:41:14.786340+0000","last_peered":"2024-03-04T15:41:14.786340+0000","last_clean":"2024-03-04T15:41:14.786340+0000","last_became_active":"2024-03-04T15:40:49.759587+0000","last_became_peered":"2024-03-04T15:40:49.759587+0000","last_unstale":"2024-03-04T15:41:14.786340+0000","last_undegraded":"2024-03-04T15:41:14.786340+0000","last_fullsized":"2024-03-04T15:41:14.786340+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:25:51.757990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785389+0000","last_change":"2024-03-04T15:40:45.737287+0000","last_active":"2024-03-04T15:41:14.785389+0000","last_peered":"2024-03-04T15:41:14.785389+0000","last_clean":"2024-03-04T15:41:14.785389+0000","last_became_active":"2024-03-04T15:40:45.736767+0000","last_became_peered":"2024-03-04T15:40:45.736767+0000","last_unstale":"2024-03-04T15:41:14.785389+0000","last_undegraded":"2024-03-04T15:41:14.785389+0000","last_fullsized":"2024-03-04T15:41:14.785389+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:40:56.125474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"31'4","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784436+0000","last_change":"2024-03-04T15:40:47.743611+0000","last_active":"2024-03-04T15:41:14.784436+0000","last_peered":"2024-03-04T15:41:14.784436+0000","last_clean":"2024-03-04T15:41:14.784436+0000","last_became_active":"2024-03-04T15:40:47.742215+0000","last_became_peered":"2024-03-04T15:40:47.742215+0000","last_unstale":"2024-03-04T15:41:14.784436+0000","last_undegraded":"2024-03-04T15:41:14.784436+0000","last_fullsized":"2024-03-04T15:41:14.784436+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:36:59.132491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786093+0000","last_change":"2024-03-04T15:40:51.770017+0000","last_active":"2024-03-04T15:41:14.786093+0000","last_peered":"2024-03-04T15:41:14.786093+0000","last_clean":"2024-03-04T15:41:14.786093+0000","last_became_active":"2024-03-04T15:40:51.769630+0000","last_became_peered":"2024-03-04T15:40:51.769630+0000","last_unstale":"2024-03-04T15:41:14.786093+0000","last_undegraded":"2024-03-04T15:41:14.786093+0000","last_fullsized":"2024-03-04T15:41:14.786093+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:19:10.304926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785304+0000","last_change":"2024-03-04T15:40:49.759655+0000","last_active":"2024-03-04T15:41:14.785304+0000","last_peered":"2024-03-04T15:41:14.785304+0000","last_clean":"2024-03-04T15:41:14.785304+0000","last_became_active":"2024-03-04T15:40:49.759272+0000","last_became_peered":"2024-03-04T15:40:49.759272+0000","last_unstale":"2024-03-04T15:41:14.785304+0000","last_undegraded":"2024-03-04T15:41:14.785304+0000","last_fullsized":"2024-03-04T15:41:14.785304+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:50:18.059827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788364+0000","last_change":"2024-03-04T15:40:45.737034+0000","last_active":"2024-03-04T15:41:14.788364+0000","last_peered":"2024-03-04T15:41:14.788364+0000","last_clean":"2024-03-04T15:41:14.788364+0000","last_became_active":"2024-03-04T15:40:45.736008+0000","last_became_peered":"2024-03-04T15:40:45.736008+0000","last_unstale":"2024-03-04T15:41:14.788364+0000","last_undegraded":"2024-03-04T15:41:14.788364+0000","last_fullsized":"2024-03-04T15:41:14.788364+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:38:35.361436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789046+0000","last_change":"2024-03-04T15:40:47.740044+0000","last_active":"2024-03-04T15:41:14.789046+0000","last_peered":"2024-03-04T15:41:14.789046+0000","last_clean":"2024-03-04T15:41:14.789046+0000","last_became_active":"2024-03-04T15:40:47.739729+0000","last_became_peered":"2024-03-04T15:40:47.739729+0000","last_unstale":"2024-03-04T15:41:14.789046+0000","last_undegraded":"2024-03-04T15:41:14.789046+0000","last_fullsized":"2024-03-04T15:41:14.789046+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:50:04.403884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.790686+0000","last_change":"2024-03-04T15:40:51.769706+0000","last_active":"2024-03-04T15:41:14.790686+0000","last_peered":"2024-03-04T15:41:14.790686+0000","last_clean":"2024-03-04T15:41:14.790686+0000","last_became_active":"2024-03-04T15:40:51.768687+0000","last_became_peered":"2024-03-04T15:40:51.768687+0000","last_unstale":"2024-03-04T15:41:14.790686+0000","last_undegraded":"2024-03-04T15:41:14.790686+0000","last_fullsized":"2024-03-04T15:41:14.790686+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:45.911392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789916+0000","last_change":"2024-03-04T15:40:45.735467+0000","last_active":"2024-03-04T15:41:14.789916+0000","last_peered":"2024-03-04T15:41:14.789916+0000","last_clean":"2024-03-04T15:41:14.789916+0000","last_became_active":"2024-03-04T15:40:45.735167+0000","last_became_peered":"2024-03-04T15:40:45.735167+0000","last_unstale":"2024-03-04T15:41:14.789916+0000","last_undegraded":"2024-03-04T15:41:14.789916+0000","last_fullsized":"2024-03-04T15:41:14.789916+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:28:03.892260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.791119+0000","last_change":"2024-03-04T15:40:47.742053+0000","last_active":"2024-03-04T15:41:14.791119+0000","last_peered":"2024-03-04T15:41:14.791119+0000","last_clean":"2024-03-04T15:41:14.791119+0000","last_became_active":"2024-03-04T15:40:47.741532+0000","last_became_peered":"2024-03-04T15:40:47.741532+0000","last_unstale":"2024-03-04T15:41:14.791119+0000","last_undegraded":"2024-03-04T15:41:14.791119+0000","last_fullsized":"2024-03-04T15:41:14.791119+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:36:53.983343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:25.782809+0000","last_change":"2024-03-04T15:40:49.757584+0000","last_active":"2024-03-04T15:41:25.782809+0000","last_peered":"2024-03-04T15:41:25.782809+0000","last_clean":"2024-03-04T15:41:25.782809+0000","last_became_active":"2024-03-04T15:40:49.757287+0000","last_became_peered":"2024-03-04T15:40:49.757287+0000","last_unstale":"2024-03-04T15:41:25.782809+0000","last_undegraded":"2024-03-04T15:41:25.782809+0000","last_fullsized":"2024-03-04T15:41:25.782809+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:22:25.682448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784614+0000","last_change":"2024-03-04T15:40:51.764770+0000","last_active":"2024-03-04T15:41:14.784614+0000","last_peered":"2024-03-04T15:41:14.784614+0000","last_clean":"2024-03-04T15:41:14.784614+0000","last_became_active":"2024-03-04T15:40:51.763162+0000","last_became_peered":"2024-03-04T15:40:51.763162+0000","last_unstale":"2024-03-04T15:41:14.784614+0000","last_undegraded":"2024-03-04T15:41:14.784614+0000","last_fullsized":"2024-03-04T15:41:14.784614+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:26.517129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787294+0000","last_change":"2024-03-04T15:40:45.733367+0000","last_active":"2024-03-04T15:41:14.787294+0000","last_peered":"2024-03-04T15:41:14.787294+0000","last_clean":"2024-03-04T15:41:14.787294+0000","last_became_active":"2024-03-04T15:40:45.733071+0000","last_became_peered":"2024-03-04T15:40:45.733071+0000","last_unstale":"2024-03-04T15:41:14.787294+0000","last_undegraded":"2024-03-04T15:41:14.787294+0000","last_fullsized":"2024-03-04T15:41:14.787294+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:26.716149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'16","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:25.782629+0000","last_change":"2024-03-04T15:40:47.742969+0000","last_active":"2024-03-04T15:41:25.782629+0000","last_peered":"2024-03-04T15:41:25.782629+0000","last_clean":"2024-03-04T15:41:25.782629+0000","last_became_active":"2024-03-04T15:40:47.742565+0000","last_became_peered":"2024-03-04T15:40:47.742565+0000","last_unstale":"2024-03-04T15:41:25.782629+0000","last_undegraded":"2024-03-04T15:41:25.782629+0000","last_fullsized":"2024-03-04T15:41:25.782629+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:00:17.427475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788561+0000","last_change":"2024-03-04T15:40:49.752370+0000","last_active":"2024-03-04T15:41:14.788561+0000","last_peered":"2024-03-04T15:41:14.788561+0000","last_clean":"2024-03-04T15:41:14.788561+0000","last_became_active":"2024-03-04T15:40:49.752006+0000","last_became_peered":"2024-03-04T15:40:49.752006+0000","last_unstale":"2024-03-04T15:41:14.788561+0000","last_undegraded":"2024-03-04T15:41:14.788561+0000","last_fullsized":"2024-03-04T15:41:14.788561+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:37:26.772434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785151+0000","last_change":"2024-03-04T15:40:51.767204+0000","last_active":"2024-03-04T15:41:14.785151+0000","last_peered":"2024-03-04T15:41:14.785151+0000","last_clean":"2024-03-04T15:41:14.785151+0000","last_became_active":"2024-03-04T15:40:51.766802+0000","last_became_peered":"2024-03-04T15:40:51.766802+0000","last_unstale":"2024-03-04T15:41:14.785151+0000","last_undegraded":"2024-03-04T15:41:14.785151+0000","last_fullsized":"2024-03-04T15:41:14.785151+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:13:56.507938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785589+0000","last_change":"2024-03-04T15:40:45.734552+0000","last_active":"2024-03-04T15:41:14.785589+0000","last_peered":"2024-03-04T15:41:14.785589+0000","last_clean":"2024-03-04T15:41:14.785589+0000","last_became_active":"2024-03-04T15:40:45.733854+0000","last_became_peered":"2024-03-04T15:40:45.733854+0000","last_unstale":"2024-03-04T15:41:14.785589+0000","last_undegraded":"2024-03-04T15:41:14.785589+0000","last_fullsized":"2024-03-04T15:41:14.785589+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:09:50.477864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'19","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786155+0000","last_change":"2024-03-04T15:40:47.742527+0000","last_active":"2024-03-04T15:41:14.786155+0000","last_peered":"2024-03-04T15:41:14.786155+0000","last_clean":"2024-03-04T15:41:14.786155+0000","last_became_active":"2024-03-04T15:40:47.741455+0000","last_became_peered":"2024-03-04T15:40:47.741455+0000","last_unstale":"2024-03-04T15:41:14.786155+0000","last_undegraded":"2024-03-04T15:41:14.786155+0000","last_fullsized":"2024-03-04T15:41:14.786155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:53:19.419399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787083+0000","last_change":"2024-03-04T15:40:49.758712+0000","last_active":"2024-03-04T15:41:14.787083+0000","last_peered":"2024-03-04T15:41:14.787083+0000","last_clean":"2024-03-04T15:41:14.787083+0000","last_became_active":"2024-03-04T15:40:49.758197+0000","last_became_peered":"2024-03-04T15:40:49.758197+0000","last_unstale":"2024-03-04T15:41:14.787083+0000","last_undegraded":"2024-03-04T15:41:14.787083+0000","last_fullsized":"2024-03-04T15:41:14.787083+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:06:23.090684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785779+0000","last_change":"2024-03-04T15:40:51.770398+0000","last_active":"2024-03-04T15:41:14.785779+0000","last_peered":"2024-03-04T15:41:14.785779+0000","last_clean":"2024-03-04T15:41:14.785779+0000","last_became_active":"2024-03-04T15:40:51.769676+0000","last_became_peered":"2024-03-04T15:40:51.769676+0000","last_unstale":"2024-03-04T15:41:14.785779+0000","last_undegraded":"2024-03-04T15:41:14.785779+0000","last_fullsized":"2024-03-04T15:41:14.785779+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:15:52.549706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"31'18","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788053+0000","last_change":"2024-03-04T15:40:47.742139+0000","last_active":"2024-03-04T15:41:14.788053+0000","last_peered":"2024-03-04T15:41:14.788053+0000","last_clean":"2024-03-04T15:41:14.788053+0000","last_became_active":"2024-03-04T15:40:47.741593+0000","last_became_peered":"2024-03-04T15:40:47.741593+0000","last_unstale":"2024-03-04T15:41:14.788053+0000","last_undegraded":"2024-03-04T15:41:14.788053+0000","last_fullsized":"2024-03-04T15:41:14.788053+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:34:28.922754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783790+0000","last_change":"2024-03-04T15:40:45.735743+0000","last_active":"2024-03-04T15:41:14.783790+0000","last_peered":"2024-03-04T15:41:14.783790+0000","last_clean":"2024-03-04T15:41:14.783790+0000","last_became_active":"2024-03-04T15:40:45.734552+0000","last_became_peered":"2024-03-04T15:40:45.734552+0000","last_unstale":"2024-03-04T15:41:14.783790+0000","last_undegraded":"2024-03-04T15:41:14.783790+0000","last_fullsized":"2024-03-04T15:41:14.783790+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:59.007985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786813+0000","last_change":"2024-03-04T15:40:49.760221+0000","last_active":"2024-03-04T15:41:14.786813+0000","last_peered":"2024-03-04T15:41:14.786813+0000","last_clean":"2024-03-04T15:41:14.786813+0000","last_became_active":"2024-03-04T15:40:49.759023+0000","last_became_peered":"2024-03-04T15:40:49.759023+0000","last_unstale":"2024-03-04T15:41:14.786813+0000","last_undegraded":"2024-03-04T15:41:14.786813+0000","last_fullsized":"2024-03-04T15:41:14.786813+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:45:11.447907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789435+0000","last_change":"2024-03-04T15:40:51.769579+0000","last_active":"2024-03-04T15:41:14.789435+0000","last_peered":"2024-03-04T15:41:14.789435+0000","last_clean":"2024-03-04T15:41:14.789435+0000","last_became_active":"2024-03-04T15:40:51.768437+0000","last_became_peered":"2024-03-04T15:40:51.768437+0000","last_unstale":"2024-03-04T15:41:14.789435+0000","last_undegraded":"2024-03-04T15:41:14.789435+0000","last_fullsized":"2024-03-04T15:41:14.789435+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:21:37.485447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784161+0000","last_change":"2024-03-04T15:40:47.740696+0000","last_active":"2024-03-04T15:41:14.784161+0000","last_peered":"2024-03-04T15:41:14.784161+0000","last_clean":"2024-03-04T15:41:14.784161+0000","last_became_active":"2024-03-04T15:40:47.739456+0000","last_became_peered":"2024-03-04T15:40:47.739456+0000","last_unstale":"2024-03-04T15:41:14.784161+0000","last_undegraded":"2024-03-04T15:41:14.784161+0000","last_fullsized":"2024-03-04T15:41:14.784161+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:51:04.116307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786012+0000","last_change":"2024-03-04T15:40:45.738203+0000","last_active":"2024-03-04T15:41:14.786012+0000","last_peered":"2024-03-04T15:41:14.786012+0000","last_clean":"2024-03-04T15:41:14.786012+0000","last_became_active":"2024-03-04T15:40:45.737854+0000","last_became_peered":"2024-03-04T15:40:45.737854+0000","last_unstale":"2024-03-04T15:41:14.786012+0000","last_undegraded":"2024-03-04T15:41:14.786012+0000","last_fullsized":"2024-03-04T15:41:14.786012+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:54:23.675588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"32'208","reported_seq":314,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:16.883708+0000","last_change":"2024-03-04T15:40:32.204382+0000","last_active":"2024-03-04T15:41:16.883708+0000","last_peered":"2024-03-04T15:41:16.883708+0000","last_clean":"2024-03-04T15:41:16.883708+0000","last_became_active":"2024-03-04T15:40:32.201194+0000","last_became_peered":"2024-03-04T15:40:32.201194+0000","last_unstale":"2024-03-04T15:41:16.883708+0000","last_undegraded":"2024-03-04T15:41:16.883708+0000","last_fullsized":"2024-03-04T15:41:16.883708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:31.186603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:31.186603+0000","last_clean_scrub_stamp":"2024-03-04T15:40:31.186603+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:00:23.712657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786348+0000","last_change":"2024-03-04T15:40:49.754783+0000","last_active":"2024-03-04T15:41:14.786348+0000","last_peered":"2024-03-04T15:41:14.786348+0000","last_clean":"2024-03-04T15:41:14.786348+0000","last_became_active":"2024-03-04T15:40:49.752916+0000","last_became_peered":"2024-03-04T15:40:49.752916+0000","last_unstale":"2024-03-04T15:41:14.786348+0000","last_undegraded":"2024-03-04T15:41:14.786348+0000","last_fullsized":"2024-03-04T15:41:14.786348+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:09:44.334354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786173+0000","last_change":"2024-03-04T15:40:51.766635+0000","last_active":"2024-03-04T15:41:14.786173+0000","last_peered":"2024-03-04T15:41:14.786173+0000","last_clean":"2024-03-04T15:41:14.786173+0000","last_became_active":"2024-03-04T15:40:51.766102+0000","last_became_peered":"2024-03-04T15:40:51.766102+0000","last_unstale":"2024-03-04T15:41:14.786173+0000","last_undegraded":"2024-03-04T15:41:14.786173+0000","last_fullsized":"2024-03-04T15:41:14.786173+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:01:40.131299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'14","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783983+0000","last_change":"2024-03-04T15:40:47.741208+0000","last_active":"2024-03-04T15:41:14.783983+0000","last_peered":"2024-03-04T15:41:14.783983+0000","last_clean":"2024-03-04T15:41:14.783983+0000","last_became_active":"2024-03-04T15:40:47.740855+0000","last_became_peered":"2024-03-04T15:40:47.740855+0000","last_unstale":"2024-03-04T15:41:14.783983+0000","last_undegraded":"2024-03-04T15:41:14.783983+0000","last_fullsized":"2024-03-04T15:41:14.783983+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:30:49.324510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784420+0000","last_change":"2024-03-04T15:40:45.740546+0000","last_active":"2024-03-04T15:41:14.784420+0000","last_peered":"2024-03-04T15:41:14.784420+0000","last_clean":"2024-03-04T15:41:14.784420+0000","last_became_active":"2024-03-04T15:40:45.739492+0000","last_became_peered":"2024-03-04T15:40:45.739492+0000","last_unstale":"2024-03-04T15:41:14.784420+0000","last_undegraded":"2024-03-04T15:41:14.784420+0000","last_fullsized":"2024-03-04T15:41:14.784420+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:04:07.515917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.790690+0000","last_change":"2024-03-04T15:40:49.752664+0000","last_active":"2024-03-04T15:41:14.790690+0000","last_peered":"2024-03-04T15:41:14.790690+0000","last_clean":"2024-03-04T15:41:14.790690+0000","last_became_active":"2024-03-04T15:40:49.752261+0000","last_became_peered":"2024-03-04T15:40:49.752261+0000","last_unstale":"2024-03-04T15:41:14.790690+0000","last_undegraded":"2024-03-04T15:41:14.790690+0000","last_fullsized":"2024-03-04T15:41:14.790690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:09:32.861079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786961+0000","last_change":"2024-03-04T15:40:51.762202+0000","last_active":"2024-03-04T15:41:14.786961+0000","last_peered":"2024-03-04T15:41:14.786961+0000","last_clean":"2024-03-04T15:41:14.786961+0000","last_became_active":"2024-03-04T15:40:51.761357+0000","last_became_peered":"2024-03-04T15:40:51.761357+0000","last_unstale":"2024-03-04T15:41:14.786961+0000","last_undegraded":"2024-03-04T15:41:14.786961+0000","last_fullsized":"2024-03-04T15:41:14.786961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:23:10.351734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.791026+0000","last_change":"2024-03-04T15:40:45.733087+0000","last_active":"2024-03-04T15:41:14.791026+0000","last_peered":"2024-03-04T15:41:14.791026+0000","last_clean":"2024-03-04T15:41:14.791026+0000","last_became_active":"2024-03-04T15:40:45.732772+0000","last_became_peered":"2024-03-04T15:40:45.732772+0000","last_unstale":"2024-03-04T15:41:14.791026+0000","last_undegraded":"2024-03-04T15:41:14.791026+0000","last_fullsized":"2024-03-04T15:41:14.791026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:12:13.317175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'28","reported_seq":81,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:25.782446+0000","last_change":"2024-03-04T15:40:47.740706+0000","last_active":"2024-03-04T15:41:25.782446+0000","last_peered":"2024-03-04T15:41:25.782446+0000","last_clean":"2024-03-04T15:41:25.782446+0000","last_became_active":"2024-03-04T15:40:47.740200+0000","last_became_peered":"2024-03-04T15:40:47.740200+0000","last_unstale":"2024-03-04T15:41:25.782446+0000","last_undegraded":"2024-03-04T15:41:25.782446+0000","last_fullsized":"2024-03-04T15:41:25.782446+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:39:44.177527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:25.782750+0000","last_change":"2024-03-04T15:40:49.757844+0000","last_active":"2024-03-04T15:41:25.782750+0000","last_peered":"2024-03-04T15:41:25.782750+0000","last_clean":"2024-03-04T15:41:25.782750+0000","last_became_active":"2024-03-04T15:40:49.757468+0000","last_became_peered":"2024-03-04T15:40:49.757468+0000","last_unstale":"2024-03-04T15:41:25.782750+0000","last_undegraded":"2024-03-04T15:41:25.782750+0000","last_fullsized":"2024-03-04T15:41:25.782750+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:26:55.339895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784318+0000","last_change":"2024-03-04T15:40:51.768462+0000","last_active":"2024-03-04T15:41:14.784318+0000","last_peered":"2024-03-04T15:41:14.784318+0000","last_clean":"2024-03-04T15:41:14.784318+0000","last_became_active":"2024-03-04T15:40:51.768190+0000","last_became_peered":"2024-03-04T15:40:51.768190+0000","last_unstale":"2024-03-04T15:41:14.784318+0000","last_undegraded":"2024-03-04T15:41:14.784318+0000","last_fullsized":"2024-03-04T15:41:14.784318+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:30:48.438436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785522+0000","last_change":"2024-03-04T15:40:45.735597+0000","last_active":"2024-03-04T15:41:14.785522+0000","last_peered":"2024-03-04T15:41:14.785522+0000","last_clean":"2024-03-04T15:41:14.785522+0000","last_became_active":"2024-03-04T15:40:45.734916+0000","last_became_peered":"2024-03-04T15:40:45.734916+0000","last_unstale":"2024-03-04T15:41:14.785522+0000","last_undegraded":"2024-03-04T15:41:14.785522+0000","last_fullsized":"2024-03-04T15:41:14.785522+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:20:58.896830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"31'13","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787115+0000","last_change":"2024-03-04T15:40:47.741295+0000","last_active":"2024-03-04T15:41:14.787115+0000","last_peered":"2024-03-04T15:41:14.787115+0000","last_clean":"2024-03-04T15:41:14.787115+0000","last_became_active":"2024-03-04T15:40:47.740939+0000","last_became_peered":"2024-03-04T15:40:47.740939+0000","last_unstale":"2024-03-04T15:41:14.787115+0000","last_undegraded":"2024-03-04T15:41:14.787115+0000","last_fullsized":"2024-03-04T15:41:14.787115+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:39:39.479581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:25.782352+0000","last_change":"2024-03-04T15:40:49.761048+0000","last_active":"2024-03-04T15:41:25.782352+0000","last_peered":"2024-03-04T15:41:25.782352+0000","last_clean":"2024-03-04T15:41:25.782352+0000","last_became_active":"2024-03-04T15:40:49.760475+0000","last_became_peered":"2024-03-04T15:40:49.760475+0000","last_unstale":"2024-03-04T15:41:25.782352+0000","last_undegraded":"2024-03-04T15:41:25.782352+0000","last_fullsized":"2024-03-04T15:41:25.782352+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:40:46.258039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.790582+0000","last_change":"2024-03-04T15:40:51.769672+0000","last_active":"2024-03-04T15:41:14.790582+0000","last_peered":"2024-03-04T15:41:14.790582+0000","last_clean":"2024-03-04T15:41:14.790582+0000","last_became_active":"2024-03-04T15:40:51.769335+0000","last_became_peered":"2024-03-04T15:40:51.769335+0000","last_unstale":"2024-03-04T15:41:14.790582+0000","last_undegraded":"2024-03-04T15:41:14.790582+0000","last_fullsized":"2024-03-04T15:41:14.790582+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:33:23.590872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786536+0000","last_change":"2024-03-04T15:40:45.737386+0000","last_active":"2024-03-04T15:41:14.786536+0000","last_peered":"2024-03-04T15:41:14.786536+0000","last_clean":"2024-03-04T15:41:14.786536+0000","last_became_active":"2024-03-04T15:40:45.736996+0000","last_became_peered":"2024-03-04T15:40:45.736996+0000","last_unstale":"2024-03-04T15:41:14.786536+0000","last_undegraded":"2024-03-04T15:41:14.786536+0000","last_fullsized":"2024-03-04T15:41:14.786536+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:31:37.304126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"31'12","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785407+0000","last_change":"2024-03-04T15:40:47.740684+0000","last_active":"2024-03-04T15:41:14.785407+0000","last_peered":"2024-03-04T15:41:14.785407+0000","last_clean":"2024-03-04T15:41:14.785407+0000","last_became_active":"2024-03-04T15:40:47.740242+0000","last_became_peered":"2024-03-04T15:40:47.740242+0000","last_unstale":"2024-03-04T15:41:14.785407+0000","last_undegraded":"2024-03-04T15:41:14.785407+0000","last_fullsized":"2024-03-04T15:41:14.785407+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:26.954241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788366+0000","last_change":"2024-03-04T15:40:49.752937+0000","last_active":"2024-03-04T15:41:14.788366+0000","last_peered":"2024-03-04T15:41:14.788366+0000","last_clean":"2024-03-04T15:41:14.788366+0000","last_became_active":"2024-03-04T15:40:49.752648+0000","last_became_peered":"2024-03-04T15:40:49.752648+0000","last_unstale":"2024-03-04T15:41:14.788366+0000","last_undegraded":"2024-03-04T15:41:14.788366+0000","last_fullsized":"2024-03-04T15:41:14.788366+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:01:05.502936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786252+0000","last_change":"2024-03-04T15:40:51.766191+0000","last_active":"2024-03-04T15:41:14.786252+0000","last_peered":"2024-03-04T15:41:14.786252+0000","last_clean":"2024-03-04T15:41:14.786252+0000","last_became_active":"2024-03-04T15:40:51.764523+0000","last_became_peered":"2024-03-04T15:40:51.764523+0000","last_unstale":"2024-03-04T15:41:14.786252+0000","last_undegraded":"2024-03-04T15:41:14.786252+0000","last_fullsized":"2024-03-04T15:41:14.786252+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:18:32.534817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:25.782584+0000","last_change":"2024-03-04T15:40:49.754778+0000","last_active":"2024-03-04T15:41:25.782584+0000","last_peered":"2024-03-04T15:41:25.782584+0000","last_clean":"2024-03-04T15:41:25.782584+0000","last_became_active":"2024-03-04T15:40:49.754280+0000","last_became_peered":"2024-03-04T15:40:49.754280+0000","last_unstale":"2024-03-04T15:41:25.782584+0000","last_undegraded":"2024-03-04T15:41:25.782584+0000","last_fullsized":"2024-03-04T15:41:25.782584+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:15:17.916931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784327+0000","last_change":"2024-03-04T15:40:45.737331+0000","last_active":"2024-03-04T15:41:14.784327+0000","last_peered":"2024-03-04T15:41:14.784327+0000","last_clean":"2024-03-04T15:41:14.784327+0000","last_became_active":"2024-03-04T15:40:45.736827+0000","last_became_peered":"2024-03-04T15:40:45.736827+0000","last_unstale":"2024-03-04T15:41:14.784327+0000","last_undegraded":"2024-03-04T15:41:14.784327+0000","last_fullsized":"2024-03-04T15:41:14.784327+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:58:55.210941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788714+0000","last_change":"2024-03-04T15:40:47.739972+0000","last_active":"2024-03-04T15:41:14.788714+0000","last_peered":"2024-03-04T15:41:14.788714+0000","last_clean":"2024-03-04T15:41:14.788714+0000","last_became_active":"2024-03-04T15:40:47.739670+0000","last_became_peered":"2024-03-04T15:40:47.739670+0000","last_unstale":"2024-03-04T15:41:14.788714+0000","last_undegraded":"2024-03-04T15:41:14.788714+0000","last_fullsized":"2024-03-04T15:41:14.788714+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:42:02.356644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.790803+0000","last_change":"2024-03-04T15:40:51.769262+0000","last_active":"2024-03-04T15:41:14.790803+0000","last_peered":"2024-03-04T15:41:14.790803+0000","last_clean":"2024-03-04T15:41:14.790803+0000","last_became_active":"2024-03-04T15:40:51.768499+0000","last_became_peered":"2024-03-04T15:40:51.768499+0000","last_unstale":"2024-03-04T15:41:14.790803+0000","last_undegraded":"2024-03-04T15:41:14.790803+0000","last_fullsized":"2024-03-04T15:41:14.790803+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:41:56.840484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786420+0000","last_change":"2024-03-04T15:40:49.759616+0000","last_active":"2024-03-04T15:41:14.786420+0000","last_peered":"2024-03-04T15:41:14.786420+0000","last_clean":"2024-03-04T15:41:14.786420+0000","last_became_active":"2024-03-04T15:40:49.758714+0000","last_became_peered":"2024-03-04T15:40:49.758714+0000","last_unstale":"2024-03-04T15:41:14.786420+0000","last_undegraded":"2024-03-04T15:41:14.786420+0000","last_fullsized":"2024-03-04T15:41:14.786420+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:40:00.663999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783585+0000","last_change":"2024-03-04T15:40:45.735926+0000","last_active":"2024-03-04T15:41:14.783585+0000","last_peered":"2024-03-04T15:41:14.783585+0000","last_clean":"2024-03-04T15:41:14.783585+0000","last_became_active":"2024-03-04T15:40:45.735400+0000","last_became_peered":"2024-03-04T15:40:45.735400+0000","last_unstale":"2024-03-04T15:41:14.783585+0000","last_undegraded":"2024-03-04T15:41:14.783585+0000","last_fullsized":"2024-03-04T15:41:14.783585+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:56:26.615544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788405+0000","last_change":"2024-03-04T15:40:47.740281+0000","last_active":"2024-03-04T15:41:14.788405+0000","last_peered":"2024-03-04T15:41:14.788405+0000","last_clean":"2024-03-04T15:41:14.788405+0000","last_became_active":"2024-03-04T15:40:47.739691+0000","last_became_peered":"2024-03-04T15:40:47.739691+0000","last_unstale":"2024-03-04T15:41:14.788405+0000","last_undegraded":"2024-03-04T15:41:14.788405+0000","last_fullsized":"2024-03-04T15:41:14.788405+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:18:11.713213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.790610+0000","last_change":"2024-03-04T15:40:51.769512+0000","last_active":"2024-03-04T15:41:14.790610+0000","last_peered":"2024-03-04T15:41:14.790610+0000","last_clean":"2024-03-04T15:41:14.790610+0000","last_became_active":"2024-03-04T15:40:51.768874+0000","last_became_peered":"2024-03-04T15:40:51.768874+0000","last_unstale":"2024-03-04T15:41:14.790610+0000","last_undegraded":"2024-03-04T15:41:14.790610+0000","last_fullsized":"2024-03-04T15:41:14.790610+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:32:36.165836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789162+0000","last_change":"2024-03-04T15:40:45.735494+0000","last_active":"2024-03-04T15:41:14.789162+0000","last_peered":"2024-03-04T15:41:14.789162+0000","last_clean":"2024-03-04T15:41:14.789162+0000","last_became_active":"2024-03-04T15:40:45.735155+0000","last_became_peered":"2024-03-04T15:40:45.735155+0000","last_unstale":"2024-03-04T15:41:14.789162+0000","last_undegraded":"2024-03-04T15:41:14.789162+0000","last_fullsized":"2024-03-04T15:41:14.789162+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:33:49.251366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785202+0000","last_change":"2024-03-04T15:40:47.739299+0000","last_active":"2024-03-04T15:41:14.785202+0000","last_peered":"2024-03-04T15:41:14.785202+0000","last_clean":"2024-03-04T15:41:14.785202+0000","last_became_active":"2024-03-04T15:40:47.738910+0000","last_became_peered":"2024-03-04T15:40:47.738910+0000","last_unstale":"2024-03-04T15:41:14.785202+0000","last_undegraded":"2024-03-04T15:41:14.785202+0000","last_fullsized":"2024-03-04T15:41:14.785202+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:57:22.219470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785908+0000","last_change":"2024-03-04T15:40:49.756320+0000","last_active":"2024-03-04T15:41:14.785908+0000","last_peered":"2024-03-04T15:41:14.785908+0000","last_clean":"2024-03-04T15:41:14.785908+0000","last_became_active":"2024-03-04T15:40:49.755969+0000","last_became_peered":"2024-03-04T15:40:49.755969+0000","last_unstale":"2024-03-04T15:41:14.785908+0000","last_undegraded":"2024-03-04T15:41:14.785908+0000","last_fullsized":"2024-03-04T15:41:14.785908+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:16:26.539391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787266+0000","last_change":"2024-03-04T15:40:51.769660+0000","last_active":"2024-03-04T15:41:14.787266+0000","last_peered":"2024-03-04T15:41:14.787266+0000","last_clean":"2024-03-04T15:41:14.787266+0000","last_became_active":"2024-03-04T15:40:51.768648+0000","last_became_peered":"2024-03-04T15:40:51.768648+0000","last_unstale":"2024-03-04T15:41:14.787266+0000","last_undegraded":"2024-03-04T15:41:14.787266+0000","last_fullsized":"2024-03-04T15:41:14.787266+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:20:55.287335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"31'8","reported_seq":37,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:25.782588+0000","last_change":"2024-03-04T15:40:49.754839+0000","last_active":"2024-03-04T15:41:25.782588+0000","last_peered":"2024-03-04T15:41:25.782588+0000","last_clean":"2024-03-04T15:41:25.782588+0000","last_became_active":"2024-03-04T15:40:49.752532+0000","last_became_peered":"2024-03-04T15:40:49.752532+0000","last_unstale":"2024-03-04T15:41:25.782588+0000","last_undegraded":"2024-03-04T15:41:25.782588+0000","last_fullsized":"2024-03-04T15:41:25.782588+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:46:59.866289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789051+0000","last_change":"2024-03-04T15:40:45.733961+0000","last_active":"2024-03-04T15:41:14.789051+0000","last_peered":"2024-03-04T15:41:14.789051+0000","last_clean":"2024-03-04T15:41:14.789051+0000","last_became_active":"2024-03-04T15:40:45.733529+0000","last_became_peered":"2024-03-04T15:40:45.733529+0000","last_unstale":"2024-03-04T15:41:14.789051+0000","last_undegraded":"2024-03-04T15:41:14.789051+0000","last_fullsized":"2024-03-04T15:41:14.789051+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:03:53.129668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'19","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785361+0000","last_change":"2024-03-04T15:40:47.742451+0000","last_active":"2024-03-04T15:41:14.785361+0000","last_peered":"2024-03-04T15:41:14.785361+0000","last_clean":"2024-03-04T15:41:14.785361+0000","last_became_active":"2024-03-04T15:40:47.741861+0000","last_became_peered":"2024-03-04T15:40:47.741861+0000","last_unstale":"2024-03-04T15:41:14.785361+0000","last_undegraded":"2024-03-04T15:41:14.785361+0000","last_fullsized":"2024-03-04T15:41:14.785361+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:29:21.105665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788488+0000","last_change":"2024-03-04T15:40:51.768299+0000","last_active":"2024-03-04T15:41:14.788488+0000","last_peered":"2024-03-04T15:41:14.788488+0000","last_clean":"2024-03-04T15:41:14.788488+0000","last_became_active":"2024-03-04T15:40:51.767982+0000","last_became_peered":"2024-03-04T15:40:51.767982+0000","last_unstale":"2024-03-04T15:41:14.788488+0000","last_undegraded":"2024-03-04T15:41:14.788488+0000","last_fullsized":"2024-03-04T15:41:14.788488+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:16:36.122393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789795+0000","last_change":"2024-03-04T15:40:45.735836+0000","last_active":"2024-03-04T15:41:14.789795+0000","last_peered":"2024-03-04T15:41:14.789795+0000","last_clean":"2024-03-04T15:41:14.789795+0000","last_became_active":"2024-03-04T15:40:45.735362+0000","last_became_peered":"2024-03-04T15:40:45.735362+0000","last_unstale":"2024-03-04T15:41:14.789795+0000","last_undegraded":"2024-03-04T15:41:14.789795+0000","last_fullsized":"2024-03-04T15:41:14.789795+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:33.692098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'17","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787205+0000","last_change":"2024-03-04T15:40:47.742947+0000","last_active":"2024-03-04T15:41:14.787205+0000","last_peered":"2024-03-04T15:41:14.787205+0000","last_clean":"2024-03-04T15:41:14.787205+0000","last_became_active":"2024-03-04T15:40:47.742132+0000","last_became_peered":"2024-03-04T15:40:47.742132+0000","last_unstale":"2024-03-04T15:41:14.787205+0000","last_undegraded":"2024-03-04T15:41:14.787205+0000","last_fullsized":"2024-03-04T15:41:14.787205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:30:24.768011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784716+0000","last_change":"2024-03-04T15:40:49.756593+0000","last_active":"2024-03-04T15:41:14.784716+0000","last_peered":"2024-03-04T15:41:14.784716+0000","last_clean":"2024-03-04T15:41:14.784716+0000","last_became_active":"2024-03-04T15:40:49.752984+0000","last_became_peered":"2024-03-04T15:40:49.752984+0000","last_unstale":"2024-03-04T15:41:14.784716+0000","last_undegraded":"2024-03-04T15:41:14.784716+0000","last_fullsized":"2024-03-04T15:41:14.784716+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:36:10.914315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783549+0000","last_change":"2024-03-04T15:40:51.765071+0000","last_active":"2024-03-04T15:41:14.783549+0000","last_peered":"2024-03-04T15:41:14.783549+0000","last_clean":"2024-03-04T15:41:14.783549+0000","last_became_active":"2024-03-04T15:40:51.763679+0000","last_became_peered":"2024-03-04T15:40:51.763679+0000","last_unstale":"2024-03-04T15:41:14.783549+0000","last_undegraded":"2024-03-04T15:41:14.783549+0000","last_fullsized":"2024-03-04T15:41:14.783549+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:41:07.548803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.790864+0000","last_change":"2024-03-04T15:40:45.733200+0000","last_active":"2024-03-04T15:41:14.790864+0000","last_peered":"2024-03-04T15:41:14.790864+0000","last_clean":"2024-03-04T15:41:14.790864+0000","last_became_active":"2024-03-04T15:40:45.732814+0000","last_became_peered":"2024-03-04T15:40:45.732814+0000","last_unstale":"2024-03-04T15:41:14.790864+0000","last_undegraded":"2024-03-04T15:41:14.790864+0000","last_fullsized":"2024-03-04T15:41:14.790864+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:38:52.744952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787222+0000","last_change":"2024-03-04T15:40:47.743481+0000","last_active":"2024-03-04T15:41:14.787222+0000","last_peered":"2024-03-04T15:41:14.787222+0000","last_clean":"2024-03-04T15:41:14.787222+0000","last_became_active":"2024-03-04T15:40:47.742836+0000","last_became_peered":"2024-03-04T15:40:47.742836+0000","last_unstale":"2024-03-04T15:41:14.787222+0000","last_undegraded":"2024-03-04T15:41:14.787222+0000","last_fullsized":"2024-03-04T15:41:14.787222+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:21.082802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784711+0000","last_change":"2024-03-04T15:40:49.759144+0000","last_active":"2024-03-04T15:41:14.784711+0000","last_peered":"2024-03-04T15:41:14.784711+0000","last_clean":"2024-03-04T15:41:14.784711+0000","last_became_active":"2024-03-04T15:40:49.758126+0000","last_became_peered":"2024-03-04T15:40:49.758126+0000","last_unstale":"2024-03-04T15:41:14.784711+0000","last_undegraded":"2024-03-04T15:41:14.784711+0000","last_fullsized":"2024-03-04T15:41:14.784711+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:42:44.534692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784430+0000","last_change":"2024-03-04T15:40:51.762601+0000","last_active":"2024-03-04T15:41:14.784430+0000","last_peered":"2024-03-04T15:41:14.784430+0000","last_clean":"2024-03-04T15:41:14.784430+0000","last_became_active":"2024-03-04T15:40:51.762064+0000","last_became_peered":"2024-03-04T15:40:51.762064+0000","last_unstale":"2024-03-04T15:41:14.784430+0000","last_undegraded":"2024-03-04T15:41:14.784430+0000","last_fullsized":"2024-03-04T15:41:14.784430+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:30:50.526073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785920+0000","last_change":"2024-03-04T15:40:45.733534+0000","last_active":"2024-03-04T15:41:14.785920+0000","last_peered":"2024-03-04T15:41:14.785920+0000","last_clean":"2024-03-04T15:41:14.785920+0000","last_became_active":"2024-03-04T15:40:45.733162+0000","last_became_peered":"2024-03-04T15:40:45.733162+0000","last_unstale":"2024-03-04T15:41:14.785920+0000","last_undegraded":"2024-03-04T15:41:14.785920+0000","last_fullsized":"2024-03-04T15:41:14.785920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:43:20.475611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787019+0000","last_change":"2024-03-04T15:40:47.742033+0000","last_active":"2024-03-04T15:41:14.787019+0000","last_peered":"2024-03-04T15:41:14.787019+0000","last_clean":"2024-03-04T15:41:14.787019+0000","last_became_active":"2024-03-04T15:40:47.741705+0000","last_became_peered":"2024-03-04T15:40:47.741705+0000","last_unstale":"2024-03-04T15:41:14.787019+0000","last_undegraded":"2024-03-04T15:41:14.787019+0000","last_fullsized":"2024-03-04T15:41:14.787019+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:23:40.235430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788630+0000","last_change":"2024-03-04T15:40:49.758940+0000","last_active":"2024-03-04T15:41:14.788630+0000","last_peered":"2024-03-04T15:41:14.788630+0000","last_clean":"2024-03-04T15:41:14.788630+0000","last_became_active":"2024-03-04T15:40:49.758640+0000","last_became_peered":"2024-03-04T15:40:49.758640+0000","last_unstale":"2024-03-04T15:41:14.788630+0000","last_undegraded":"2024-03-04T15:41:14.788630+0000","last_fullsized":"2024-03-04T15:41:14.788630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:47:41.637803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411343,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738394,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T15:41:35.934 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T15:41:36.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:35 smithi005 ceph-mon[25858]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:36.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:35 smithi087 ceph-mon[27664]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:36.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:35 smithi080 ceph-mon[27564]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:36.524 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:37.357 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:36 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: ::ffff:172.21.15.80 - - [04/Mar/2024:15:41:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T15:41:38.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:37 smithi005 ceph-mon[25858]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:38.181 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:41:38 smithi005 podman[49204]: 2024-03-04 15:41:38.142138654 +0000 UTC m=+0.446018130 container create f2f4e735e3ca8b7364dc6d9215b79b42ec15a4943e7c14003eb53c3b2a7d2761 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T15:41:38.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:37 smithi087 ceph-mon[27664]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:38.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:37 smithi080 ceph-mon[27564]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:38.755 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:41:38 smithi005 podman[49204]: 2024-03-04 15:41:38.444502467 +0000 UTC m=+0.748381943 container init f2f4e735e3ca8b7364dc6d9215b79b42ec15a4943e7c14003eb53c3b2a7d2761 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T15:41:38.755 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:41:38 smithi005 podman[49204]: 2024-03-04 15:41:38.450435016 +0000 UTC m=+0.754314492 container start f2f4e735e3ca8b7364dc6d9215b79b42ec15a4943e7c14003eb53c3b2a7d2761 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T15:41:38.755 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:41:38 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a[49242]: ts=2024-03-04T15:41:38.464Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-04T15:41:38.756 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:41:38 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a[49242]: ts=2024-03-04T15:41:38.464Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-04T15:41:38.756 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:41:38 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a[49242]: ts=2024-03-04T15:41:38.464Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.5 port=9094 2024-03-04T15:41:38.756 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:41:38 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a[49242]: ts=2024-03-04T15:41:38.465Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-04T15:41:38.756 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:41:38 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a[49242]: ts=2024-03-04T15:41:38.489Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-04T15:41:38.756 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:41:38 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a[49242]: ts=2024-03-04T15:41:38.489Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-04T15:41:38.756 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:41:38 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a[49242]: ts=2024-03-04T15:41:38.492Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-04T15:41:38.756 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:41:38 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a[49242]: ts=2024-03-04T15:41:38.492Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-04T15:41:38.756 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:41:38 smithi005 bash[49204]: f2f4e735e3ca8b7364dc6d9215b79b42ec15a4943e7c14003eb53c3b2a7d2761 2024-03-04T15:41:38.756 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:41:38 smithi005 systemd[1]: Started Ceph alertmanager.a for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:41:39.016 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:38 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:41:38] ENGINE Bus STOPPING 2024-03-04T15:41:39.016 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:38 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:41:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T15:41:39.016 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:38 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:41:38] ENGINE Bus STOPPED 2024-03-04T15:41:39.016 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:38 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:41:38] ENGINE Bus STARTING 2024-03-04T15:41:39.016 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:38 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:41:38] ENGINE Serving on http://:::9283 2024-03-04T15:41:39.016 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:38 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:41:38] ENGINE Bus STARTED 2024-03-04T15:41:39.021 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:41:39.023 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-03-04T15:41:39.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:39 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:39.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:39 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:39.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:39 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:39.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:39 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:39.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:39 smithi087 ceph-mon[27664]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-04T15:41:39.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:39 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:39.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:39 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:39.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:39 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T15:41:39.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:39 smithi087 ceph-mon[27664]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T15:41:39.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:39 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:39.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:39 smithi087 ceph-mon[27664]: Deploying daemon grafana.a on smithi080 2024-03-04T15:41:39.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:39 smithi087 ceph-mon[27664]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:39.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:39 smithi087 ceph-mon[27664]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:41:39.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:39 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:39.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:39 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:39.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:39 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:39.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:39 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:39.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:39 smithi005 ceph-mon[25858]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-04T15:41:39.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:39 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:39.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:39 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:39.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:39 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T15:41:39.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:39 smithi005 ceph-mon[25858]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T15:41:39.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:39 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:39.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:39 smithi005 ceph-mon[25858]: Deploying daemon grafana.a on smithi080 2024-03-04T15:41:39.872 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:39 smithi005 ceph-mon[25858]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:39.873 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:39 smithi005 ceph-mon[25858]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:41:39.902 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-03-04T15:41:38.799499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":96636,"kb_used_data":3108,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":281126532,"statfs":{"total":287972524032,"available":287873568768,"internally_reserved":0,"allocated":3182592,"data_stored":2109504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003447"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785025+0000","last_change":"2024-03-04T15:40:51.769959+0000","last_active":"2024-03-04T15:41:14.785025+0000","last_peered":"2024-03-04T15:41:14.785025+0000","last_clean":"2024-03-04T15:41:14.785025+0000","last_became_active":"2024-03-04T15:40:51.769564+0000","last_became_peered":"2024-03-04T15:40:51.769564+0000","last_unstale":"2024-03-04T15:41:14.785025+0000","last_undegraded":"2024-03-04T15:41:14.785025+0000","last_fullsized":"2024-03-04T15:41:14.785025+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:13:03.634445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788256+0000","last_change":"2024-03-04T15:40:49.757616+0000","last_active":"2024-03-04T15:41:14.788256+0000","last_peered":"2024-03-04T15:41:14.788256+0000","last_clean":"2024-03-04T15:41:14.788256+0000","last_became_active":"2024-03-04T15:40:49.757327+0000","last_became_peered":"2024-03-04T15:40:49.757327+0000","last_unstale":"2024-03-04T15:41:14.788256+0000","last_undegraded":"2024-03-04T15:41:14.788256+0000","last_fullsized":"2024-03-04T15:41:14.788256+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:00:53.575547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786670+0000","last_change":"2024-03-04T15:40:45.735989+0000","last_active":"2024-03-04T15:41:14.786670+0000","last_peered":"2024-03-04T15:41:14.786670+0000","last_clean":"2024-03-04T15:41:14.786670+0000","last_became_active":"2024-03-04T15:40:45.735018+0000","last_became_peered":"2024-03-04T15:40:45.735018+0000","last_unstale":"2024-03-04T15:41:14.786670+0000","last_undegraded":"2024-03-04T15:41:14.786670+0000","last_fullsized":"2024-03-04T15:41:14.786670+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:26:56.697354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786032+0000","last_change":"2024-03-04T15:40:47.740686+0000","last_active":"2024-03-04T15:41:14.786032+0000","last_peered":"2024-03-04T15:41:14.786032+0000","last_clean":"2024-03-04T15:41:14.786032+0000","last_became_active":"2024-03-04T15:40:47.740005+0000","last_became_peered":"2024-03-04T15:40:47.740005+0000","last_unstale":"2024-03-04T15:41:14.786032+0000","last_undegraded":"2024-03-04T15:41:14.786032+0000","last_fullsized":"2024-03-04T15:41:14.786032+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:13:06.487583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783462+0000","last_change":"2024-03-04T15:40:51.765009+0000","last_active":"2024-03-04T15:41:14.783462+0000","last_peered":"2024-03-04T15:41:14.783462+0000","last_clean":"2024-03-04T15:41:14.783462+0000","last_became_active":"2024-03-04T15:40:51.764154+0000","last_became_peered":"2024-03-04T15:40:51.764154+0000","last_unstale":"2024-03-04T15:41:14.783462+0000","last_undegraded":"2024-03-04T15:41:14.783462+0000","last_fullsized":"2024-03-04T15:41:14.783462+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:24:32.866079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784748+0000","last_change":"2024-03-04T15:40:49.756733+0000","last_active":"2024-03-04T15:41:14.784748+0000","last_peered":"2024-03-04T15:41:14.784748+0000","last_clean":"2024-03-04T15:41:14.784748+0000","last_became_active":"2024-03-04T15:40:49.756337+0000","last_became_peered":"2024-03-04T15:40:49.756337+0000","last_unstale":"2024-03-04T15:41:14.784748+0000","last_undegraded":"2024-03-04T15:41:14.784748+0000","last_fullsized":"2024-03-04T15:41:14.784748+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:47:18.124798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785685+0000","last_change":"2024-03-04T15:40:45.735614+0000","last_active":"2024-03-04T15:41:14.785685+0000","last_peered":"2024-03-04T15:41:14.785685+0000","last_clean":"2024-03-04T15:41:14.785685+0000","last_became_active":"2024-03-04T15:40:45.734940+0000","last_became_peered":"2024-03-04T15:40:45.734940+0000","last_unstale":"2024-03-04T15:41:14.785685+0000","last_undegraded":"2024-03-04T15:41:14.785685+0000","last_fullsized":"2024-03-04T15:41:14.785685+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:45:36.911780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786502+0000","last_change":"2024-03-04T15:40:47.740459+0000","last_active":"2024-03-04T15:41:14.786502+0000","last_peered":"2024-03-04T15:41:14.786502+0000","last_clean":"2024-03-04T15:41:14.786502+0000","last_became_active":"2024-03-04T15:40:47.739339+0000","last_became_peered":"2024-03-04T15:40:47.739339+0000","last_unstale":"2024-03-04T15:41:14.786502+0000","last_undegraded":"2024-03-04T15:41:14.786502+0000","last_fullsized":"2024-03-04T15:41:14.786502+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:48:25.179538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788275+0000","last_change":"2024-03-04T15:40:51.767954+0000","last_active":"2024-03-04T15:41:14.788275+0000","last_peered":"2024-03-04T15:41:14.788275+0000","last_clean":"2024-03-04T15:41:14.788275+0000","last_became_active":"2024-03-04T15:40:51.767645+0000","last_became_peered":"2024-03-04T15:40:51.767645+0000","last_unstale":"2024-03-04T15:41:14.788275+0000","last_undegraded":"2024-03-04T15:41:14.788275+0000","last_fullsized":"2024-03-04T15:41:14.788275+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:32:46.739642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784899+0000","last_change":"2024-03-04T15:40:49.757496+0000","last_active":"2024-03-04T15:41:14.784899+0000","last_peered":"2024-03-04T15:41:14.784899+0000","last_clean":"2024-03-04T15:41:14.784899+0000","last_became_active":"2024-03-04T15:40:49.757186+0000","last_became_peered":"2024-03-04T15:40:49.757186+0000","last_unstale":"2024-03-04T15:41:14.784899+0000","last_undegraded":"2024-03-04T15:41:14.784899+0000","last_fullsized":"2024-03-04T15:41:14.784899+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:12:38.250653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.791351+0000","last_change":"2024-03-04T15:40:45.735621+0000","last_active":"2024-03-04T15:41:14.791351+0000","last_peered":"2024-03-04T15:41:14.791351+0000","last_clean":"2024-03-04T15:41:14.791351+0000","last_became_active":"2024-03-04T15:40:45.734748+0000","last_became_peered":"2024-03-04T15:40:45.734748+0000","last_unstale":"2024-03-04T15:41:14.791351+0000","last_undegraded":"2024-03-04T15:41:14.791351+0000","last_fullsized":"2024-03-04T15:41:14.791351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:11.985815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786207+0000","last_change":"2024-03-04T15:40:47.743651+0000","last_active":"2024-03-04T15:41:14.786207+0000","last_peered":"2024-03-04T15:41:14.786207+0000","last_clean":"2024-03-04T15:41:14.786207+0000","last_became_active":"2024-03-04T15:40:47.743147+0000","last_became_peered":"2024-03-04T15:40:47.743147+0000","last_unstale":"2024-03-04T15:41:14.786207+0000","last_undegraded":"2024-03-04T15:41:14.786207+0000","last_fullsized":"2024-03-04T15:41:14.786207+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:21.900919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784721+0000","last_change":"2024-03-04T15:40:51.764182+0000","last_active":"2024-03-04T15:41:14.784721+0000","last_peered":"2024-03-04T15:41:14.784721+0000","last_clean":"2024-03-04T15:41:14.784721+0000","last_became_active":"2024-03-04T15:40:51.762173+0000","last_became_peered":"2024-03-04T15:40:51.762173+0000","last_unstale":"2024-03-04T15:41:14.784721+0000","last_undegraded":"2024-03-04T15:41:14.784721+0000","last_fullsized":"2024-03-04T15:41:14.784721+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:38:07.307651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784535+0000","last_change":"2024-03-04T15:40:49.756262+0000","last_active":"2024-03-04T15:41:14.784535+0000","last_peered":"2024-03-04T15:41:14.784535+0000","last_clean":"2024-03-04T15:41:14.784535+0000","last_became_active":"2024-03-04T15:40:49.753202+0000","last_became_peered":"2024-03-04T15:40:49.753202+0000","last_unstale":"2024-03-04T15:41:14.784535+0000","last_undegraded":"2024-03-04T15:41:14.784535+0000","last_fullsized":"2024-03-04T15:41:14.784535+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:32:41.115965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789700+0000","last_change":"2024-03-04T15:40:45.736529+0000","last_active":"2024-03-04T15:41:14.789700+0000","last_peered":"2024-03-04T15:41:14.789700+0000","last_clean":"2024-03-04T15:41:14.789700+0000","last_became_active":"2024-03-04T15:40:45.735582+0000","last_became_peered":"2024-03-04T15:40:45.735582+0000","last_unstale":"2024-03-04T15:41:14.789700+0000","last_undegraded":"2024-03-04T15:41:14.789700+0000","last_fullsized":"2024-03-04T15:41:14.789700+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:11:33.370775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.791253+0000","last_change":"2024-03-04T15:40:47.745175+0000","last_active":"2024-03-04T15:41:14.791253+0000","last_peered":"2024-03-04T15:41:14.791253+0000","last_clean":"2024-03-04T15:41:14.791253+0000","last_became_active":"2024-03-04T15:40:47.744868+0000","last_became_peered":"2024-03-04T15:40:47.744868+0000","last_unstale":"2024-03-04T15:41:14.791253+0000","last_undegraded":"2024-03-04T15:41:14.791253+0000","last_fullsized":"2024-03-04T15:41:14.791253+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:30:16.660951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785792+0000","last_change":"2024-03-04T15:40:51.762535+0000","last_active":"2024-03-04T15:41:14.785792+0000","last_peered":"2024-03-04T15:41:14.785792+0000","last_clean":"2024-03-04T15:41:14.785792+0000","last_became_active":"2024-03-04T15:40:51.762198+0000","last_became_peered":"2024-03-04T15:40:51.762198+0000","last_unstale":"2024-03-04T15:41:14.785792+0000","last_undegraded":"2024-03-04T15:41:14.785792+0000","last_fullsized":"2024-03-04T15:41:14.785792+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:33:53.857497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.790706+0000","last_change":"2024-03-04T15:40:49.758011+0000","last_active":"2024-03-04T15:41:14.790706+0000","last_peered":"2024-03-04T15:41:14.790706+0000","last_clean":"2024-03-04T15:41:14.790706+0000","last_became_active":"2024-03-04T15:40:49.757654+0000","last_became_peered":"2024-03-04T15:40:49.757654+0000","last_unstale":"2024-03-04T15:41:14.790706+0000","last_undegraded":"2024-03-04T15:41:14.790706+0000","last_fullsized":"2024-03-04T15:41:14.790706+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:20:37.542151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789211+0000","last_change":"2024-03-04T15:40:45.736433+0000","last_active":"2024-03-04T15:41:14.789211+0000","last_peered":"2024-03-04T15:41:14.789211+0000","last_clean":"2024-03-04T15:41:14.789211+0000","last_became_active":"2024-03-04T15:40:45.735672+0000","last_became_peered":"2024-03-04T15:40:45.735672+0000","last_unstale":"2024-03-04T15:41:14.789211+0000","last_undegraded":"2024-03-04T15:41:14.789211+0000","last_fullsized":"2024-03-04T15:41:14.789211+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:33:12.025991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789261+0000","last_change":"2024-03-04T15:40:47.741747+0000","last_active":"2024-03-04T15:41:14.789261+0000","last_peered":"2024-03-04T15:41:14.789261+0000","last_clean":"2024-03-04T15:41:14.789261+0000","last_became_active":"2024-03-04T15:40:47.741435+0000","last_became_peered":"2024-03-04T15:40:47.741435+0000","last_unstale":"2024-03-04T15:41:14.789261+0000","last_undegraded":"2024-03-04T15:41:14.789261+0000","last_fullsized":"2024-03-04T15:41:14.789261+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:52:19.519700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785676+0000","last_change":"2024-03-04T15:40:51.769036+0000","last_active":"2024-03-04T15:41:14.785676+0000","last_peered":"2024-03-04T15:41:14.785676+0000","last_clean":"2024-03-04T15:41:14.785676+0000","last_became_active":"2024-03-04T15:40:51.768615+0000","last_became_peered":"2024-03-04T15:40:51.768615+0000","last_unstale":"2024-03-04T15:41:14.785676+0000","last_undegraded":"2024-03-04T15:41:14.785676+0000","last_fullsized":"2024-03-04T15:41:14.785676+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:01:05.970415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789824+0000","last_change":"2024-03-04T15:40:49.752517+0000","last_active":"2024-03-04T15:41:14.789824+0000","last_peered":"2024-03-04T15:41:14.789824+0000","last_clean":"2024-03-04T15:41:14.789824+0000","last_became_active":"2024-03-04T15:40:49.752161+0000","last_became_peered":"2024-03-04T15:40:49.752161+0000","last_unstale":"2024-03-04T15:41:14.789824+0000","last_undegraded":"2024-03-04T15:41:14.789824+0000","last_fullsized":"2024-03-04T15:41:14.789824+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:00:42.835911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785723+0000","last_change":"2024-03-04T15:40:45.734647+0000","last_active":"2024-03-04T15:41:14.785723+0000","last_peered":"2024-03-04T15:41:14.785723+0000","last_clean":"2024-03-04T15:41:14.785723+0000","last_became_active":"2024-03-04T15:40:45.733972+0000","last_became_peered":"2024-03-04T15:40:45.733972+0000","last_unstale":"2024-03-04T15:41:14.785723+0000","last_undegraded":"2024-03-04T15:41:14.785723+0000","last_fullsized":"2024-03-04T15:41:14.785723+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:21:37.130794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'5","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788933+0000","last_change":"2024-03-04T15:40:47.741957+0000","last_active":"2024-03-04T15:41:14.788933+0000","last_peered":"2024-03-04T15:41:14.788933+0000","last_clean":"2024-03-04T15:41:14.788933+0000","last_became_active":"2024-03-04T15:40:47.741458+0000","last_became_peered":"2024-03-04T15:40:47.741458+0000","last_unstale":"2024-03-04T15:41:14.788933+0000","last_undegraded":"2024-03-04T15:41:14.788933+0000","last_fullsized":"2024-03-04T15:41:14.788933+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:32:26.007477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784898+0000","last_change":"2024-03-04T15:40:51.769872+0000","last_active":"2024-03-04T15:41:14.784898+0000","last_peered":"2024-03-04T15:41:14.784898+0000","last_clean":"2024-03-04T15:41:14.784898+0000","last_became_active":"2024-03-04T15:40:51.769389+0000","last_became_peered":"2024-03-04T15:40:51.769389+0000","last_unstale":"2024-03-04T15:41:14.784898+0000","last_undegraded":"2024-03-04T15:41:14.784898+0000","last_fullsized":"2024-03-04T15:41:14.784898+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:46:49.503639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'8","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:30.783920+0000","last_change":"2024-03-04T15:40:49.758544+0000","last_active":"2024-03-04T15:41:30.783920+0000","last_peered":"2024-03-04T15:41:30.783920+0000","last_clean":"2024-03-04T15:41:30.783920+0000","last_became_active":"2024-03-04T15:40:49.757583+0000","last_became_peered":"2024-03-04T15:40:49.757583+0000","last_unstale":"2024-03-04T15:41:30.783920+0000","last_undegraded":"2024-03-04T15:41:30.783920+0000","last_fullsized":"2024-03-04T15:41:30.783920+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:15:49.658483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783886+0000","last_change":"2024-03-04T15:40:45.736001+0000","last_active":"2024-03-04T15:41:14.783886+0000","last_peered":"2024-03-04T15:41:14.783886+0000","last_clean":"2024-03-04T15:41:14.783886+0000","last_became_active":"2024-03-04T15:40:45.735635+0000","last_became_peered":"2024-03-04T15:40:45.735635+0000","last_unstale":"2024-03-04T15:41:14.783886+0000","last_undegraded":"2024-03-04T15:41:14.783886+0000","last_fullsized":"2024-03-04T15:41:14.783886+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:48:24.241284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783342+0000","last_change":"2024-03-04T15:40:47.741660+0000","last_active":"2024-03-04T15:41:14.783342+0000","last_peered":"2024-03-04T15:41:14.783342+0000","last_clean":"2024-03-04T15:41:14.783342+0000","last_became_active":"2024-03-04T15:40:47.740867+0000","last_became_peered":"2024-03-04T15:40:47.740867+0000","last_unstale":"2024-03-04T15:41:14.783342+0000","last_undegraded":"2024-03-04T15:41:14.783342+0000","last_fullsized":"2024-03-04T15:41:14.783342+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:43:52.782567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786067+0000","last_change":"2024-03-04T15:40:51.766417+0000","last_active":"2024-03-04T15:41:14.786067+0000","last_peered":"2024-03-04T15:41:14.786067+0000","last_clean":"2024-03-04T15:41:14.786067+0000","last_became_active":"2024-03-04T15:40:51.764751+0000","last_became_peered":"2024-03-04T15:40:51.764751+0000","last_unstale":"2024-03-04T15:41:14.786067+0000","last_undegraded":"2024-03-04T15:41:14.786067+0000","last_fullsized":"2024-03-04T15:41:14.786067+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:21:11.249731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785201+0000","last_change":"2024-03-04T15:40:49.757575+0000","last_active":"2024-03-04T15:41:14.785201+0000","last_peered":"2024-03-04T15:41:14.785201+0000","last_clean":"2024-03-04T15:41:14.785201+0000","last_became_active":"2024-03-04T15:40:49.757172+0000","last_became_peered":"2024-03-04T15:40:49.757172+0000","last_unstale":"2024-03-04T15:41:14.785201+0000","last_undegraded":"2024-03-04T15:41:14.785201+0000","last_fullsized":"2024-03-04T15:41:14.785201+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:29.152753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788299+0000","last_change":"2024-03-04T15:40:45.737162+0000","last_active":"2024-03-04T15:41:14.788299+0000","last_peered":"2024-03-04T15:41:14.788299+0000","last_clean":"2024-03-04T15:41:14.788299+0000","last_became_active":"2024-03-04T15:40:45.736401+0000","last_became_peered":"2024-03-04T15:40:45.736401+0000","last_unstale":"2024-03-04T15:41:14.788299+0000","last_undegraded":"2024-03-04T15:41:14.788299+0000","last_fullsized":"2024-03-04T15:41:14.788299+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:27:34.287422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783430+0000","last_change":"2024-03-04T15:40:47.741535+0000","last_active":"2024-03-04T15:41:14.783430+0000","last_peered":"2024-03-04T15:41:14.783430+0000","last_clean":"2024-03-04T15:41:14.783430+0000","last_became_active":"2024-03-04T15:40:47.741016+0000","last_became_peered":"2024-03-04T15:40:47.741016+0000","last_unstale":"2024-03-04T15:41:14.783430+0000","last_undegraded":"2024-03-04T15:41:14.783430+0000","last_fullsized":"2024-03-04T15:41:14.783430+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:23:53.943756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784954+0000","last_change":"2024-03-04T15:40:51.769971+0000","last_active":"2024-03-04T15:41:14.784954+0000","last_peered":"2024-03-04T15:41:14.784954+0000","last_clean":"2024-03-04T15:41:14.784954+0000","last_became_active":"2024-03-04T15:40:51.769628+0000","last_became_peered":"2024-03-04T15:40:51.769628+0000","last_unstale":"2024-03-04T15:41:14.784954+0000","last_undegraded":"2024-03-04T15:41:14.784954+0000","last_fullsized":"2024-03-04T15:41:14.784954+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:47:50.538893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783345+0000","last_change":"2024-03-04T15:40:49.760118+0000","last_active":"2024-03-04T15:41:14.783345+0000","last_peered":"2024-03-04T15:41:14.783345+0000","last_clean":"2024-03-04T15:41:14.783345+0000","last_became_active":"2024-03-04T15:40:49.759629+0000","last_became_peered":"2024-03-04T15:40:49.759629+0000","last_unstale":"2024-03-04T15:41:14.783345+0000","last_undegraded":"2024-03-04T15:41:14.783345+0000","last_fullsized":"2024-03-04T15:41:14.783345+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:53:49.102411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787089+0000","last_change":"2024-03-04T15:40:45.739273+0000","last_active":"2024-03-04T15:41:14.787089+0000","last_peered":"2024-03-04T15:41:14.787089+0000","last_clean":"2024-03-04T15:41:14.787089+0000","last_became_active":"2024-03-04T15:40:45.738854+0000","last_became_peered":"2024-03-04T15:40:45.738854+0000","last_unstale":"2024-03-04T15:41:14.787089+0000","last_undegraded":"2024-03-04T15:41:14.787089+0000","last_fullsized":"2024-03-04T15:41:14.787089+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:57:03.191782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786057+0000","last_change":"2024-03-04T15:40:47.742402+0000","last_active":"2024-03-04T15:41:14.786057+0000","last_peered":"2024-03-04T15:41:14.786057+0000","last_clean":"2024-03-04T15:41:14.786057+0000","last_became_active":"2024-03-04T15:40:47.741747+0000","last_became_peered":"2024-03-04T15:40:47.741747+0000","last_unstale":"2024-03-04T15:41:14.786057+0000","last_undegraded":"2024-03-04T15:41:14.786057+0000","last_fullsized":"2024-03-04T15:41:14.786057+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:53:37.578353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785013+0000","last_change":"2024-03-04T15:40:51.762576+0000","last_active":"2024-03-04T15:41:14.785013+0000","last_peered":"2024-03-04T15:41:14.785013+0000","last_clean":"2024-03-04T15:41:14.785013+0000","last_became_active":"2024-03-04T15:40:51.762104+0000","last_became_peered":"2024-03-04T15:40:51.762104+0000","last_unstale":"2024-03-04T15:41:14.785013+0000","last_undegraded":"2024-03-04T15:41:14.785013+0000","last_fullsized":"2024-03-04T15:41:14.785013+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:47:33.290012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783647+0000","last_change":"2024-03-04T15:40:49.759706+0000","last_active":"2024-03-04T15:41:14.783647+0000","last_peered":"2024-03-04T15:41:14.783647+0000","last_clean":"2024-03-04T15:41:14.783647+0000","last_became_active":"2024-03-04T15:40:49.758356+0000","last_became_peered":"2024-03-04T15:40:49.758356+0000","last_unstale":"2024-03-04T15:41:14.783647+0000","last_undegraded":"2024-03-04T15:41:14.783647+0000","last_fullsized":"2024-03-04T15:41:14.783647+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:19:29.972687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786154+0000","last_change":"2024-03-04T15:40:45.737848+0000","last_active":"2024-03-04T15:41:14.786154+0000","last_peered":"2024-03-04T15:41:14.786154+0000","last_clean":"2024-03-04T15:41:14.786154+0000","last_became_active":"2024-03-04T15:40:45.737534+0000","last_became_peered":"2024-03-04T15:40:45.737534+0000","last_unstale":"2024-03-04T15:41:14.786154+0000","last_undegraded":"2024-03-04T15:41:14.786154+0000","last_fullsized":"2024-03-04T15:41:14.786154+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:03:46.278852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"31'6","reported_seq":32,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786899+0000","last_change":"2024-03-04T15:40:47.741220+0000","last_active":"2024-03-04T15:41:14.786899+0000","last_peered":"2024-03-04T15:41:14.786899+0000","last_clean":"2024-03-04T15:41:14.786899+0000","last_became_active":"2024-03-04T15:40:47.740500+0000","last_became_peered":"2024-03-04T15:40:47.740500+0000","last_unstale":"2024-03-04T15:41:14.786899+0000","last_undegraded":"2024-03-04T15:41:14.786899+0000","last_fullsized":"2024-03-04T15:41:14.786899+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:10:21.366771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"31'1","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788442+0000","last_change":"2024-03-04T15:40:51.769503+0000","last_active":"2024-03-04T15:41:14.788442+0000","last_peered":"2024-03-04T15:41:14.788442+0000","last_clean":"2024-03-04T15:41:14.788442+0000","last_became_active":"2024-03-04T15:40:51.768444+0000","last_became_peered":"2024-03-04T15:40:51.768444+0000","last_unstale":"2024-03-04T15:41:14.788442+0000","last_undegraded":"2024-03-04T15:41:14.788442+0000","last_fullsized":"2024-03-04T15:41:14.788442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:57:56.690862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785080+0000","last_change":"2024-03-04T15:40:49.756625+0000","last_active":"2024-03-04T15:41:14.785080+0000","last_peered":"2024-03-04T15:41:14.785080+0000","last_clean":"2024-03-04T15:41:14.785080+0000","last_became_active":"2024-03-04T15:40:49.756281+0000","last_became_peered":"2024-03-04T15:40:49.756281+0000","last_unstale":"2024-03-04T15:41:14.785080+0000","last_undegraded":"2024-03-04T15:41:14.785080+0000","last_fullsized":"2024-03-04T15:41:14.785080+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:24:52.549448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.791180+0000","last_change":"2024-03-04T15:40:45.733948+0000","last_active":"2024-03-04T15:41:14.791180+0000","last_peered":"2024-03-04T15:41:14.791180+0000","last_clean":"2024-03-04T15:41:14.791180+0000","last_became_active":"2024-03-04T15:40:45.733532+0000","last_became_peered":"2024-03-04T15:40:45.733532+0000","last_unstale":"2024-03-04T15:41:14.791180+0000","last_undegraded":"2024-03-04T15:41:14.791180+0000","last_fullsized":"2024-03-04T15:41:14.791180+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:04:15.869340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787076+0000","last_change":"2024-03-04T15:40:47.743159+0000","last_active":"2024-03-04T15:41:14.787076+0000","last_peered":"2024-03-04T15:41:14.787076+0000","last_clean":"2024-03-04T15:41:14.787076+0000","last_became_active":"2024-03-04T15:40:47.742399+0000","last_became_peered":"2024-03-04T15:40:47.742399+0000","last_unstale":"2024-03-04T15:41:14.787076+0000","last_undegraded":"2024-03-04T15:41:14.787076+0000","last_fullsized":"2024-03-04T15:41:14.787076+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:35:27.075374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788432+0000","last_change":"2024-03-04T15:40:51.769547+0000","last_active":"2024-03-04T15:41:14.788432+0000","last_peered":"2024-03-04T15:41:14.788432+0000","last_clean":"2024-03-04T15:41:14.788432+0000","last_became_active":"2024-03-04T15:40:51.768527+0000","last_became_peered":"2024-03-04T15:40:51.768527+0000","last_unstale":"2024-03-04T15:41:14.788432+0000","last_undegraded":"2024-03-04T15:41:14.788432+0000","last_fullsized":"2024-03-04T15:41:14.788432+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:24:14.841245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784041+0000","last_change":"2024-03-04T15:40:49.760421+0000","last_active":"2024-03-04T15:41:14.784041+0000","last_peered":"2024-03-04T15:41:14.784041+0000","last_clean":"2024-03-04T15:41:14.784041+0000","last_became_active":"2024-03-04T15:40:49.760045+0000","last_became_peered":"2024-03-04T15:40:49.760045+0000","last_unstale":"2024-03-04T15:41:14.784041+0000","last_undegraded":"2024-03-04T15:41:14.784041+0000","last_fullsized":"2024-03-04T15:41:14.784041+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:15:52.398290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787209+0000","last_change":"2024-03-04T15:40:45.739369+0000","last_active":"2024-03-04T15:41:14.787209+0000","last_peered":"2024-03-04T15:41:14.787209+0000","last_clean":"2024-03-04T15:41:14.787209+0000","last_became_active":"2024-03-04T15:40:45.739017+0000","last_became_peered":"2024-03-04T15:40:45.739017+0000","last_unstale":"2024-03-04T15:41:14.787209+0000","last_undegraded":"2024-03-04T15:41:14.787209+0000","last_fullsized":"2024-03-04T15:41:14.787209+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:31:48.754552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.791125+0000","last_change":"2024-03-04T15:40:47.742047+0000","last_active":"2024-03-04T15:41:14.791125+0000","last_peered":"2024-03-04T15:41:14.791125+0000","last_clean":"2024-03-04T15:41:14.791125+0000","last_became_active":"2024-03-04T15:40:47.741599+0000","last_became_peered":"2024-03-04T15:40:47.741599+0000","last_unstale":"2024-03-04T15:41:14.791125+0000","last_undegraded":"2024-03-04T15:41:14.791125+0000","last_fullsized":"2024-03-04T15:41:14.791125+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:20:49.033294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786985+0000","last_change":"2024-03-04T15:40:51.770171+0000","last_active":"2024-03-04T15:41:14.786985+0000","last_peered":"2024-03-04T15:41:14.786985+0000","last_clean":"2024-03-04T15:41:14.786985+0000","last_became_active":"2024-03-04T15:40:51.769837+0000","last_became_peered":"2024-03-04T15:40:51.769837+0000","last_unstale":"2024-03-04T15:41:14.786985+0000","last_undegraded":"2024-03-04T15:41:14.786985+0000","last_fullsized":"2024-03-04T15:41:14.786985+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:46:32.095191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:30.783891+0000","last_change":"2024-03-04T15:40:49.750132+0000","last_active":"2024-03-04T15:41:30.783891+0000","last_peered":"2024-03-04T15:41:30.783891+0000","last_clean":"2024-03-04T15:41:30.783891+0000","last_became_active":"2024-03-04T15:40:49.749836+0000","last_became_peered":"2024-03-04T15:40:49.749836+0000","last_unstale":"2024-03-04T15:41:30.783891+0000","last_undegraded":"2024-03-04T15:41:30.783891+0000","last_fullsized":"2024-03-04T15:41:30.783891+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:48:59.571636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788811+0000","last_change":"2024-03-04T15:40:45.734935+0000","last_active":"2024-03-04T15:41:14.788811+0000","last_peered":"2024-03-04T15:41:14.788811+0000","last_clean":"2024-03-04T15:41:14.788811+0000","last_became_active":"2024-03-04T15:40:45.734080+0000","last_became_peered":"2024-03-04T15:40:45.734080+0000","last_unstale":"2024-03-04T15:41:14.788811+0000","last_undegraded":"2024-03-04T15:41:14.788811+0000","last_fullsized":"2024-03-04T15:41:14.788811+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:41:10.393533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784202+0000","last_change":"2024-03-04T15:40:47.739311+0000","last_active":"2024-03-04T15:41:14.784202+0000","last_peered":"2024-03-04T15:41:14.784202+0000","last_clean":"2024-03-04T15:41:14.784202+0000","last_became_active":"2024-03-04T15:40:47.738977+0000","last_became_peered":"2024-03-04T15:40:47.738977+0000","last_unstale":"2024-03-04T15:41:14.784202+0000","last_undegraded":"2024-03-04T15:41:14.784202+0000","last_fullsized":"2024-03-04T15:41:14.784202+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:00:06.410395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785257+0000","last_change":"2024-03-04T15:40:51.763798+0000","last_active":"2024-03-04T15:41:14.785257+0000","last_peered":"2024-03-04T15:41:14.785257+0000","last_clean":"2024-03-04T15:41:14.785257+0000","last_became_active":"2024-03-04T15:40:51.762400+0000","last_became_peered":"2024-03-04T15:40:51.762400+0000","last_unstale":"2024-03-04T15:41:14.785257+0000","last_undegraded":"2024-03-04T15:41:14.785257+0000","last_fullsized":"2024-03-04T15:41:14.785257+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:55:10.737732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:30.783940+0000","last_change":"2024-03-04T15:40:49.757829+0000","last_active":"2024-03-04T15:41:30.783940+0000","last_peered":"2024-03-04T15:41:30.783940+0000","last_clean":"2024-03-04T15:41:30.783940+0000","last_became_active":"2024-03-04T15:40:49.757491+0000","last_became_peered":"2024-03-04T15:40:49.757491+0000","last_unstale":"2024-03-04T15:41:30.783940+0000","last_undegraded":"2024-03-04T15:41:30.783940+0000","last_fullsized":"2024-03-04T15:41:30.783940+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:23:17.843177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"24'1","reported_seq":33,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788899+0000","last_change":"2024-03-04T15:40:45.733121+0000","last_active":"2024-03-04T15:41:14.788899+0000","last_peered":"2024-03-04T15:41:14.788899+0000","last_clean":"2024-03-04T15:41:14.788899+0000","last_became_active":"2024-03-04T15:40:45.732811+0000","last_became_peered":"2024-03-04T15:40:45.732811+0000","last_unstale":"2024-03-04T15:41:14.788899+0000","last_undegraded":"2024-03-04T15:41:14.788899+0000","last_fullsized":"2024-03-04T15:41:14.788899+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:57.484451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785455+0000","last_change":"2024-03-04T15:40:47.742600+0000","last_active":"2024-03-04T15:41:14.785455+0000","last_peered":"2024-03-04T15:41:14.785455+0000","last_clean":"2024-03-04T15:41:14.785455+0000","last_became_active":"2024-03-04T15:40:47.741503+0000","last_became_peered":"2024-03-04T15:40:47.741503+0000","last_unstale":"2024-03-04T15:41:14.785455+0000","last_undegraded":"2024-03-04T15:41:14.785455+0000","last_fullsized":"2024-03-04T15:41:14.785455+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:06:12.243951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785721+0000","last_change":"2024-03-04T15:40:51.764833+0000","last_active":"2024-03-04T15:41:14.785721+0000","last_peered":"2024-03-04T15:41:14.785721+0000","last_clean":"2024-03-04T15:41:14.785721+0000","last_became_active":"2024-03-04T15:40:51.764487+0000","last_became_peered":"2024-03-04T15:40:51.764487+0000","last_unstale":"2024-03-04T15:41:14.785721+0000","last_undegraded":"2024-03-04T15:41:14.785721+0000","last_fullsized":"2024-03-04T15:41:14.785721+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:45:43.949495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786340+0000","last_change":"2024-03-04T15:40:49.760632+0000","last_active":"2024-03-04T15:41:14.786340+0000","last_peered":"2024-03-04T15:41:14.786340+0000","last_clean":"2024-03-04T15:41:14.786340+0000","last_became_active":"2024-03-04T15:40:49.759587+0000","last_became_peered":"2024-03-04T15:40:49.759587+0000","last_unstale":"2024-03-04T15:41:14.786340+0000","last_undegraded":"2024-03-04T15:41:14.786340+0000","last_fullsized":"2024-03-04T15:41:14.786340+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:25:51.757990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785389+0000","last_change":"2024-03-04T15:40:45.737287+0000","last_active":"2024-03-04T15:41:14.785389+0000","last_peered":"2024-03-04T15:41:14.785389+0000","last_clean":"2024-03-04T15:41:14.785389+0000","last_became_active":"2024-03-04T15:40:45.736767+0000","last_became_peered":"2024-03-04T15:40:45.736767+0000","last_unstale":"2024-03-04T15:41:14.785389+0000","last_undegraded":"2024-03-04T15:41:14.785389+0000","last_fullsized":"2024-03-04T15:41:14.785389+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:40:56.125474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"31'4","reported_seq":29,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784436+0000","last_change":"2024-03-04T15:40:47.743611+0000","last_active":"2024-03-04T15:41:14.784436+0000","last_peered":"2024-03-04T15:41:14.784436+0000","last_clean":"2024-03-04T15:41:14.784436+0000","last_became_active":"2024-03-04T15:40:47.742215+0000","last_became_peered":"2024-03-04T15:40:47.742215+0000","last_unstale":"2024-03-04T15:41:14.784436+0000","last_undegraded":"2024-03-04T15:41:14.784436+0000","last_fullsized":"2024-03-04T15:41:14.784436+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:36:59.132491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786093+0000","last_change":"2024-03-04T15:40:51.770017+0000","last_active":"2024-03-04T15:41:14.786093+0000","last_peered":"2024-03-04T15:41:14.786093+0000","last_clean":"2024-03-04T15:41:14.786093+0000","last_became_active":"2024-03-04T15:40:51.769630+0000","last_became_peered":"2024-03-04T15:40:51.769630+0000","last_unstale":"2024-03-04T15:41:14.786093+0000","last_undegraded":"2024-03-04T15:41:14.786093+0000","last_fullsized":"2024-03-04T15:41:14.786093+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:19:10.304926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785304+0000","last_change":"2024-03-04T15:40:49.759655+0000","last_active":"2024-03-04T15:41:14.785304+0000","last_peered":"2024-03-04T15:41:14.785304+0000","last_clean":"2024-03-04T15:41:14.785304+0000","last_became_active":"2024-03-04T15:40:49.759272+0000","last_became_peered":"2024-03-04T15:40:49.759272+0000","last_unstale":"2024-03-04T15:41:14.785304+0000","last_undegraded":"2024-03-04T15:41:14.785304+0000","last_fullsized":"2024-03-04T15:41:14.785304+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:50:18.059827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788364+0000","last_change":"2024-03-04T15:40:45.737034+0000","last_active":"2024-03-04T15:41:14.788364+0000","last_peered":"2024-03-04T15:41:14.788364+0000","last_clean":"2024-03-04T15:41:14.788364+0000","last_became_active":"2024-03-04T15:40:45.736008+0000","last_became_peered":"2024-03-04T15:40:45.736008+0000","last_unstale":"2024-03-04T15:41:14.788364+0000","last_undegraded":"2024-03-04T15:41:14.788364+0000","last_fullsized":"2024-03-04T15:41:14.788364+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:38:35.361436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789046+0000","last_change":"2024-03-04T15:40:47.740044+0000","last_active":"2024-03-04T15:41:14.789046+0000","last_peered":"2024-03-04T15:41:14.789046+0000","last_clean":"2024-03-04T15:41:14.789046+0000","last_became_active":"2024-03-04T15:40:47.739729+0000","last_became_peered":"2024-03-04T15:40:47.739729+0000","last_unstale":"2024-03-04T15:41:14.789046+0000","last_undegraded":"2024-03-04T15:41:14.789046+0000","last_fullsized":"2024-03-04T15:41:14.789046+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:50:04.403884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.790686+0000","last_change":"2024-03-04T15:40:51.769706+0000","last_active":"2024-03-04T15:41:14.790686+0000","last_peered":"2024-03-04T15:41:14.790686+0000","last_clean":"2024-03-04T15:41:14.790686+0000","last_became_active":"2024-03-04T15:40:51.768687+0000","last_became_peered":"2024-03-04T15:40:51.768687+0000","last_unstale":"2024-03-04T15:41:14.790686+0000","last_undegraded":"2024-03-04T15:41:14.790686+0000","last_fullsized":"2024-03-04T15:41:14.790686+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:45.911392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789916+0000","last_change":"2024-03-04T15:40:45.735467+0000","last_active":"2024-03-04T15:41:14.789916+0000","last_peered":"2024-03-04T15:41:14.789916+0000","last_clean":"2024-03-04T15:41:14.789916+0000","last_became_active":"2024-03-04T15:40:45.735167+0000","last_became_peered":"2024-03-04T15:40:45.735167+0000","last_unstale":"2024-03-04T15:41:14.789916+0000","last_undegraded":"2024-03-04T15:41:14.789916+0000","last_fullsized":"2024-03-04T15:41:14.789916+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:28:03.892260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"31'11","reported_seq":42,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.791119+0000","last_change":"2024-03-04T15:40:47.742053+0000","last_active":"2024-03-04T15:41:14.791119+0000","last_peered":"2024-03-04T15:41:14.791119+0000","last_clean":"2024-03-04T15:41:14.791119+0000","last_became_active":"2024-03-04T15:40:47.741532+0000","last_became_peered":"2024-03-04T15:40:47.741532+0000","last_unstale":"2024-03-04T15:41:14.791119+0000","last_undegraded":"2024-03-04T15:41:14.791119+0000","last_fullsized":"2024-03-04T15:41:14.791119+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:36:53.983343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:30.784009+0000","last_change":"2024-03-04T15:40:49.757584+0000","last_active":"2024-03-04T15:41:30.784009+0000","last_peered":"2024-03-04T15:41:30.784009+0000","last_clean":"2024-03-04T15:41:30.784009+0000","last_became_active":"2024-03-04T15:40:49.757287+0000","last_became_peered":"2024-03-04T15:40:49.757287+0000","last_unstale":"2024-03-04T15:41:30.784009+0000","last_undegraded":"2024-03-04T15:41:30.784009+0000","last_fullsized":"2024-03-04T15:41:30.784009+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:22:25.682448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784614+0000","last_change":"2024-03-04T15:40:51.764770+0000","last_active":"2024-03-04T15:41:14.784614+0000","last_peered":"2024-03-04T15:41:14.784614+0000","last_clean":"2024-03-04T15:41:14.784614+0000","last_became_active":"2024-03-04T15:40:51.763162+0000","last_became_peered":"2024-03-04T15:40:51.763162+0000","last_unstale":"2024-03-04T15:41:14.784614+0000","last_undegraded":"2024-03-04T15:41:14.784614+0000","last_fullsized":"2024-03-04T15:41:14.784614+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:26.517129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787294+0000","last_change":"2024-03-04T15:40:45.733367+0000","last_active":"2024-03-04T15:41:14.787294+0000","last_peered":"2024-03-04T15:41:14.787294+0000","last_clean":"2024-03-04T15:41:14.787294+0000","last_became_active":"2024-03-04T15:40:45.733071+0000","last_became_peered":"2024-03-04T15:40:45.733071+0000","last_unstale":"2024-03-04T15:41:14.787294+0000","last_undegraded":"2024-03-04T15:41:14.787294+0000","last_fullsized":"2024-03-04T15:41:14.787294+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:26.716149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"31'16","reported_seq":56,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:30.783810+0000","last_change":"2024-03-04T15:40:47.742969+0000","last_active":"2024-03-04T15:41:30.783810+0000","last_peered":"2024-03-04T15:41:30.783810+0000","last_clean":"2024-03-04T15:41:30.783810+0000","last_became_active":"2024-03-04T15:40:47.742565+0000","last_became_peered":"2024-03-04T15:40:47.742565+0000","last_unstale":"2024-03-04T15:41:30.783810+0000","last_undegraded":"2024-03-04T15:41:30.783810+0000","last_fullsized":"2024-03-04T15:41:30.783810+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:00:17.427475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788561+0000","last_change":"2024-03-04T15:40:49.752370+0000","last_active":"2024-03-04T15:41:14.788561+0000","last_peered":"2024-03-04T15:41:14.788561+0000","last_clean":"2024-03-04T15:41:14.788561+0000","last_became_active":"2024-03-04T15:40:49.752006+0000","last_became_peered":"2024-03-04T15:40:49.752006+0000","last_unstale":"2024-03-04T15:41:14.788561+0000","last_undegraded":"2024-03-04T15:41:14.788561+0000","last_fullsized":"2024-03-04T15:41:14.788561+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:37:26.772434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785151+0000","last_change":"2024-03-04T15:40:51.767204+0000","last_active":"2024-03-04T15:41:14.785151+0000","last_peered":"2024-03-04T15:41:14.785151+0000","last_clean":"2024-03-04T15:41:14.785151+0000","last_became_active":"2024-03-04T15:40:51.766802+0000","last_became_peered":"2024-03-04T15:40:51.766802+0000","last_unstale":"2024-03-04T15:41:14.785151+0000","last_undegraded":"2024-03-04T15:41:14.785151+0000","last_fullsized":"2024-03-04T15:41:14.785151+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:13:56.507938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785589+0000","last_change":"2024-03-04T15:40:45.734552+0000","last_active":"2024-03-04T15:41:14.785589+0000","last_peered":"2024-03-04T15:41:14.785589+0000","last_clean":"2024-03-04T15:41:14.785589+0000","last_became_active":"2024-03-04T15:40:45.733854+0000","last_became_peered":"2024-03-04T15:40:45.733854+0000","last_unstale":"2024-03-04T15:41:14.785589+0000","last_undegraded":"2024-03-04T15:41:14.785589+0000","last_fullsized":"2024-03-04T15:41:14.785589+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:09:50.477864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'19","reported_seq":59,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786155+0000","last_change":"2024-03-04T15:40:47.742527+0000","last_active":"2024-03-04T15:41:14.786155+0000","last_peered":"2024-03-04T15:41:14.786155+0000","last_clean":"2024-03-04T15:41:14.786155+0000","last_became_active":"2024-03-04T15:40:47.741455+0000","last_became_peered":"2024-03-04T15:40:47.741455+0000","last_unstale":"2024-03-04T15:41:14.786155+0000","last_undegraded":"2024-03-04T15:41:14.786155+0000","last_fullsized":"2024-03-04T15:41:14.786155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:53:19.419399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787083+0000","last_change":"2024-03-04T15:40:49.758712+0000","last_active":"2024-03-04T15:41:14.787083+0000","last_peered":"2024-03-04T15:41:14.787083+0000","last_clean":"2024-03-04T15:41:14.787083+0000","last_became_active":"2024-03-04T15:40:49.758197+0000","last_became_peered":"2024-03-04T15:40:49.758197+0000","last_unstale":"2024-03-04T15:41:14.787083+0000","last_undegraded":"2024-03-04T15:41:14.787083+0000","last_fullsized":"2024-03-04T15:41:14.787083+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:06:23.090684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785779+0000","last_change":"2024-03-04T15:40:51.770398+0000","last_active":"2024-03-04T15:41:14.785779+0000","last_peered":"2024-03-04T15:41:14.785779+0000","last_clean":"2024-03-04T15:41:14.785779+0000","last_became_active":"2024-03-04T15:40:51.769676+0000","last_became_peered":"2024-03-04T15:40:51.769676+0000","last_unstale":"2024-03-04T15:41:14.785779+0000","last_undegraded":"2024-03-04T15:41:14.785779+0000","last_fullsized":"2024-03-04T15:41:14.785779+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:15:52.549706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"31'18","reported_seq":55,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788053+0000","last_change":"2024-03-04T15:40:47.742139+0000","last_active":"2024-03-04T15:41:14.788053+0000","last_peered":"2024-03-04T15:41:14.788053+0000","last_clean":"2024-03-04T15:41:14.788053+0000","last_became_active":"2024-03-04T15:40:47.741593+0000","last_became_peered":"2024-03-04T15:40:47.741593+0000","last_unstale":"2024-03-04T15:41:14.788053+0000","last_undegraded":"2024-03-04T15:41:14.788053+0000","last_fullsized":"2024-03-04T15:41:14.788053+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:34:28.922754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783790+0000","last_change":"2024-03-04T15:40:45.735743+0000","last_active":"2024-03-04T15:41:14.783790+0000","last_peered":"2024-03-04T15:41:14.783790+0000","last_clean":"2024-03-04T15:41:14.783790+0000","last_became_active":"2024-03-04T15:40:45.734552+0000","last_became_peered":"2024-03-04T15:40:45.734552+0000","last_unstale":"2024-03-04T15:41:14.783790+0000","last_undegraded":"2024-03-04T15:41:14.783790+0000","last_fullsized":"2024-03-04T15:41:14.783790+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:59.007985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786813+0000","last_change":"2024-03-04T15:40:49.760221+0000","last_active":"2024-03-04T15:41:14.786813+0000","last_peered":"2024-03-04T15:41:14.786813+0000","last_clean":"2024-03-04T15:41:14.786813+0000","last_became_active":"2024-03-04T15:40:49.759023+0000","last_became_peered":"2024-03-04T15:40:49.759023+0000","last_unstale":"2024-03-04T15:41:14.786813+0000","last_undegraded":"2024-03-04T15:41:14.786813+0000","last_fullsized":"2024-03-04T15:41:14.786813+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:45:11.447907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789435+0000","last_change":"2024-03-04T15:40:51.769579+0000","last_active":"2024-03-04T15:41:14.789435+0000","last_peered":"2024-03-04T15:41:14.789435+0000","last_clean":"2024-03-04T15:41:14.789435+0000","last_became_active":"2024-03-04T15:40:51.768437+0000","last_became_peered":"2024-03-04T15:40:51.768437+0000","last_unstale":"2024-03-04T15:41:14.789435+0000","last_undegraded":"2024-03-04T15:41:14.789435+0000","last_fullsized":"2024-03-04T15:41:14.789435+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:21:37.485447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784161+0000","last_change":"2024-03-04T15:40:47.740696+0000","last_active":"2024-03-04T15:41:14.784161+0000","last_peered":"2024-03-04T15:41:14.784161+0000","last_clean":"2024-03-04T15:41:14.784161+0000","last_became_active":"2024-03-04T15:40:47.739456+0000","last_became_peered":"2024-03-04T15:40:47.739456+0000","last_unstale":"2024-03-04T15:41:14.784161+0000","last_undegraded":"2024-03-04T15:41:14.784161+0000","last_fullsized":"2024-03-04T15:41:14.784161+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:51:04.116307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786012+0000","last_change":"2024-03-04T15:40:45.738203+0000","last_active":"2024-03-04T15:41:14.786012+0000","last_peered":"2024-03-04T15:41:14.786012+0000","last_clean":"2024-03-04T15:41:14.786012+0000","last_became_active":"2024-03-04T15:40:45.737854+0000","last_became_peered":"2024-03-04T15:40:45.737854+0000","last_unstale":"2024-03-04T15:41:14.786012+0000","last_undegraded":"2024-03-04T15:41:14.786012+0000","last_fullsized":"2024-03-04T15:41:14.786012+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:54:23.675588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"32'208","reported_seq":314,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:16.883708+0000","last_change":"2024-03-04T15:40:32.204382+0000","last_active":"2024-03-04T15:41:16.883708+0000","last_peered":"2024-03-04T15:41:16.883708+0000","last_clean":"2024-03-04T15:41:16.883708+0000","last_became_active":"2024-03-04T15:40:32.201194+0000","last_became_peered":"2024-03-04T15:40:32.201194+0000","last_unstale":"2024-03-04T15:41:16.883708+0000","last_undegraded":"2024-03-04T15:41:16.883708+0000","last_fullsized":"2024-03-04T15:41:16.883708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:31.186603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:31.186603+0000","last_clean_scrub_stamp":"2024-03-04T15:40:31.186603+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:00:23.712657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786348+0000","last_change":"2024-03-04T15:40:49.754783+0000","last_active":"2024-03-04T15:41:14.786348+0000","last_peered":"2024-03-04T15:41:14.786348+0000","last_clean":"2024-03-04T15:41:14.786348+0000","last_became_active":"2024-03-04T15:40:49.752916+0000","last_became_peered":"2024-03-04T15:40:49.752916+0000","last_unstale":"2024-03-04T15:41:14.786348+0000","last_undegraded":"2024-03-04T15:41:14.786348+0000","last_fullsized":"2024-03-04T15:41:14.786348+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:09:44.334354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786173+0000","last_change":"2024-03-04T15:40:51.766635+0000","last_active":"2024-03-04T15:41:14.786173+0000","last_peered":"2024-03-04T15:41:14.786173+0000","last_clean":"2024-03-04T15:41:14.786173+0000","last_became_active":"2024-03-04T15:40:51.766102+0000","last_became_peered":"2024-03-04T15:40:51.766102+0000","last_unstale":"2024-03-04T15:41:14.786173+0000","last_undegraded":"2024-03-04T15:41:14.786173+0000","last_fullsized":"2024-03-04T15:41:14.786173+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:01:40.131299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"31'14","reported_seq":44,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783983+0000","last_change":"2024-03-04T15:40:47.741208+0000","last_active":"2024-03-04T15:41:14.783983+0000","last_peered":"2024-03-04T15:41:14.783983+0000","last_clean":"2024-03-04T15:41:14.783983+0000","last_became_active":"2024-03-04T15:40:47.740855+0000","last_became_peered":"2024-03-04T15:40:47.740855+0000","last_unstale":"2024-03-04T15:41:14.783983+0000","last_undegraded":"2024-03-04T15:41:14.783983+0000","last_fullsized":"2024-03-04T15:41:14.783983+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:30:49.324510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784420+0000","last_change":"2024-03-04T15:40:45.740546+0000","last_active":"2024-03-04T15:41:14.784420+0000","last_peered":"2024-03-04T15:41:14.784420+0000","last_clean":"2024-03-04T15:41:14.784420+0000","last_became_active":"2024-03-04T15:40:45.739492+0000","last_became_peered":"2024-03-04T15:40:45.739492+0000","last_unstale":"2024-03-04T15:41:14.784420+0000","last_undegraded":"2024-03-04T15:41:14.784420+0000","last_fullsized":"2024-03-04T15:41:14.784420+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:04:07.515917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.790690+0000","last_change":"2024-03-04T15:40:49.752664+0000","last_active":"2024-03-04T15:41:14.790690+0000","last_peered":"2024-03-04T15:41:14.790690+0000","last_clean":"2024-03-04T15:41:14.790690+0000","last_became_active":"2024-03-04T15:40:49.752261+0000","last_became_peered":"2024-03-04T15:40:49.752261+0000","last_unstale":"2024-03-04T15:41:14.790690+0000","last_undegraded":"2024-03-04T15:41:14.790690+0000","last_fullsized":"2024-03-04T15:41:14.790690+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:09:32.861079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786961+0000","last_change":"2024-03-04T15:40:51.762202+0000","last_active":"2024-03-04T15:41:14.786961+0000","last_peered":"2024-03-04T15:41:14.786961+0000","last_clean":"2024-03-04T15:41:14.786961+0000","last_became_active":"2024-03-04T15:40:51.761357+0000","last_became_peered":"2024-03-04T15:40:51.761357+0000","last_unstale":"2024-03-04T15:41:14.786961+0000","last_undegraded":"2024-03-04T15:41:14.786961+0000","last_fullsized":"2024-03-04T15:41:14.786961+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:23:10.351734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.791026+0000","last_change":"2024-03-04T15:40:45.733087+0000","last_active":"2024-03-04T15:41:14.791026+0000","last_peered":"2024-03-04T15:41:14.791026+0000","last_clean":"2024-03-04T15:41:14.791026+0000","last_became_active":"2024-03-04T15:40:45.732772+0000","last_became_peered":"2024-03-04T15:40:45.732772+0000","last_unstale":"2024-03-04T15:41:14.791026+0000","last_undegraded":"2024-03-04T15:41:14.791026+0000","last_fullsized":"2024-03-04T15:41:14.791026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:12:13.317175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"31'28","reported_seq":82,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:30.783423+0000","last_change":"2024-03-04T15:40:47.740706+0000","last_active":"2024-03-04T15:41:30.783423+0000","last_peered":"2024-03-04T15:41:30.783423+0000","last_clean":"2024-03-04T15:41:30.783423+0000","last_became_active":"2024-03-04T15:40:47.740200+0000","last_became_peered":"2024-03-04T15:40:47.740200+0000","last_unstale":"2024-03-04T15:41:30.783423+0000","last_undegraded":"2024-03-04T15:41:30.783423+0000","last_fullsized":"2024-03-04T15:41:30.783423+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:39:44.177527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:30.783899+0000","last_change":"2024-03-04T15:40:49.757844+0000","last_active":"2024-03-04T15:41:30.783899+0000","last_peered":"2024-03-04T15:41:30.783899+0000","last_clean":"2024-03-04T15:41:30.783899+0000","last_became_active":"2024-03-04T15:40:49.757468+0000","last_became_peered":"2024-03-04T15:40:49.757468+0000","last_unstale":"2024-03-04T15:41:30.783899+0000","last_undegraded":"2024-03-04T15:41:30.783899+0000","last_fullsized":"2024-03-04T15:41:30.783899+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:26:55.339895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784318+0000","last_change":"2024-03-04T15:40:51.768462+0000","last_active":"2024-03-04T15:41:14.784318+0000","last_peered":"2024-03-04T15:41:14.784318+0000","last_clean":"2024-03-04T15:41:14.784318+0000","last_became_active":"2024-03-04T15:40:51.768190+0000","last_became_peered":"2024-03-04T15:40:51.768190+0000","last_unstale":"2024-03-04T15:41:14.784318+0000","last_undegraded":"2024-03-04T15:41:14.784318+0000","last_fullsized":"2024-03-04T15:41:14.784318+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:30:48.438436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":28,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785522+0000","last_change":"2024-03-04T15:40:45.735597+0000","last_active":"2024-03-04T15:41:14.785522+0000","last_peered":"2024-03-04T15:41:14.785522+0000","last_clean":"2024-03-04T15:41:14.785522+0000","last_became_active":"2024-03-04T15:40:45.734916+0000","last_became_peered":"2024-03-04T15:40:45.734916+0000","last_unstale":"2024-03-04T15:41:14.785522+0000","last_undegraded":"2024-03-04T15:41:14.785522+0000","last_fullsized":"2024-03-04T15:41:14.785522+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:20:58.896830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"31'13","reported_seq":50,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787115+0000","last_change":"2024-03-04T15:40:47.741295+0000","last_active":"2024-03-04T15:41:14.787115+0000","last_peered":"2024-03-04T15:41:14.787115+0000","last_clean":"2024-03-04T15:41:14.787115+0000","last_became_active":"2024-03-04T15:40:47.740939+0000","last_became_peered":"2024-03-04T15:40:47.740939+0000","last_unstale":"2024-03-04T15:41:14.787115+0000","last_undegraded":"2024-03-04T15:41:14.787115+0000","last_fullsized":"2024-03-04T15:41:14.787115+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:39:39.479581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:30.783405+0000","last_change":"2024-03-04T15:40:49.761048+0000","last_active":"2024-03-04T15:41:30.783405+0000","last_peered":"2024-03-04T15:41:30.783405+0000","last_clean":"2024-03-04T15:41:30.783405+0000","last_became_active":"2024-03-04T15:40:49.760475+0000","last_became_peered":"2024-03-04T15:40:49.760475+0000","last_unstale":"2024-03-04T15:41:30.783405+0000","last_undegraded":"2024-03-04T15:41:30.783405+0000","last_fullsized":"2024-03-04T15:41:30.783405+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:40:46.258039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.790582+0000","last_change":"2024-03-04T15:40:51.769672+0000","last_active":"2024-03-04T15:41:14.790582+0000","last_peered":"2024-03-04T15:41:14.790582+0000","last_clean":"2024-03-04T15:41:14.790582+0000","last_became_active":"2024-03-04T15:40:51.769335+0000","last_became_peered":"2024-03-04T15:40:51.769335+0000","last_unstale":"2024-03-04T15:41:14.790582+0000","last_undegraded":"2024-03-04T15:41:14.790582+0000","last_fullsized":"2024-03-04T15:41:14.790582+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:33:23.590872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786536+0000","last_change":"2024-03-04T15:40:45.737386+0000","last_active":"2024-03-04T15:41:14.786536+0000","last_peered":"2024-03-04T15:41:14.786536+0000","last_clean":"2024-03-04T15:41:14.786536+0000","last_became_active":"2024-03-04T15:40:45.736996+0000","last_became_peered":"2024-03-04T15:40:45.736996+0000","last_unstale":"2024-03-04T15:41:14.786536+0000","last_undegraded":"2024-03-04T15:41:14.786536+0000","last_fullsized":"2024-03-04T15:41:14.786536+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:31:37.304126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"31'12","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785407+0000","last_change":"2024-03-04T15:40:47.740684+0000","last_active":"2024-03-04T15:41:14.785407+0000","last_peered":"2024-03-04T15:41:14.785407+0000","last_clean":"2024-03-04T15:41:14.785407+0000","last_became_active":"2024-03-04T15:40:47.740242+0000","last_became_peered":"2024-03-04T15:40:47.740242+0000","last_unstale":"2024-03-04T15:41:14.785407+0000","last_undegraded":"2024-03-04T15:41:14.785407+0000","last_fullsized":"2024-03-04T15:41:14.785407+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:26.954241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788366+0000","last_change":"2024-03-04T15:40:49.752937+0000","last_active":"2024-03-04T15:41:14.788366+0000","last_peered":"2024-03-04T15:41:14.788366+0000","last_clean":"2024-03-04T15:41:14.788366+0000","last_became_active":"2024-03-04T15:40:49.752648+0000","last_became_peered":"2024-03-04T15:40:49.752648+0000","last_unstale":"2024-03-04T15:41:14.788366+0000","last_undegraded":"2024-03-04T15:41:14.788366+0000","last_fullsized":"2024-03-04T15:41:14.788366+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:01:05.502936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786252+0000","last_change":"2024-03-04T15:40:51.766191+0000","last_active":"2024-03-04T15:41:14.786252+0000","last_peered":"2024-03-04T15:41:14.786252+0000","last_clean":"2024-03-04T15:41:14.786252+0000","last_became_active":"2024-03-04T15:40:51.764523+0000","last_became_peered":"2024-03-04T15:40:51.764523+0000","last_unstale":"2024-03-04T15:41:14.786252+0000","last_undegraded":"2024-03-04T15:41:14.786252+0000","last_fullsized":"2024-03-04T15:41:14.786252+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:18:32.534817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:30.783828+0000","last_change":"2024-03-04T15:40:49.754778+0000","last_active":"2024-03-04T15:41:30.783828+0000","last_peered":"2024-03-04T15:41:30.783828+0000","last_clean":"2024-03-04T15:41:30.783828+0000","last_became_active":"2024-03-04T15:40:49.754280+0000","last_became_peered":"2024-03-04T15:40:49.754280+0000","last_unstale":"2024-03-04T15:41:30.783828+0000","last_undegraded":"2024-03-04T15:41:30.783828+0000","last_fullsized":"2024-03-04T15:41:30.783828+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:15:17.916931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784327+0000","last_change":"2024-03-04T15:40:45.737331+0000","last_active":"2024-03-04T15:41:14.784327+0000","last_peered":"2024-03-04T15:41:14.784327+0000","last_clean":"2024-03-04T15:41:14.784327+0000","last_became_active":"2024-03-04T15:40:45.736827+0000","last_became_peered":"2024-03-04T15:40:45.736827+0000","last_unstale":"2024-03-04T15:41:14.784327+0000","last_undegraded":"2024-03-04T15:41:14.784327+0000","last_fullsized":"2024-03-04T15:41:14.784327+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:58:55.210941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"31'12","reported_seq":46,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788714+0000","last_change":"2024-03-04T15:40:47.739972+0000","last_active":"2024-03-04T15:41:14.788714+0000","last_peered":"2024-03-04T15:41:14.788714+0000","last_clean":"2024-03-04T15:41:14.788714+0000","last_became_active":"2024-03-04T15:40:47.739670+0000","last_became_peered":"2024-03-04T15:40:47.739670+0000","last_unstale":"2024-03-04T15:41:14.788714+0000","last_undegraded":"2024-03-04T15:41:14.788714+0000","last_fullsized":"2024-03-04T15:41:14.788714+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:42:02.356644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.790803+0000","last_change":"2024-03-04T15:40:51.769262+0000","last_active":"2024-03-04T15:41:14.790803+0000","last_peered":"2024-03-04T15:41:14.790803+0000","last_clean":"2024-03-04T15:41:14.790803+0000","last_became_active":"2024-03-04T15:40:51.768499+0000","last_became_peered":"2024-03-04T15:40:51.768499+0000","last_unstale":"2024-03-04T15:41:14.790803+0000","last_undegraded":"2024-03-04T15:41:14.790803+0000","last_fullsized":"2024-03-04T15:41:14.790803+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:41:56.840484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.786420+0000","last_change":"2024-03-04T15:40:49.759616+0000","last_active":"2024-03-04T15:41:14.786420+0000","last_peered":"2024-03-04T15:41:14.786420+0000","last_clean":"2024-03-04T15:41:14.786420+0000","last_became_active":"2024-03-04T15:40:49.758714+0000","last_became_peered":"2024-03-04T15:40:49.758714+0000","last_unstale":"2024-03-04T15:41:14.786420+0000","last_undegraded":"2024-03-04T15:41:14.786420+0000","last_fullsized":"2024-03-04T15:41:14.786420+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:40:00.663999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783585+0000","last_change":"2024-03-04T15:40:45.735926+0000","last_active":"2024-03-04T15:41:14.783585+0000","last_peered":"2024-03-04T15:41:14.783585+0000","last_clean":"2024-03-04T15:41:14.783585+0000","last_became_active":"2024-03-04T15:40:45.735400+0000","last_became_peered":"2024-03-04T15:40:45.735400+0000","last_unstale":"2024-03-04T15:41:14.783585+0000","last_undegraded":"2024-03-04T15:41:14.783585+0000","last_fullsized":"2024-03-04T15:41:14.783585+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:56:26.615544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788405+0000","last_change":"2024-03-04T15:40:47.740281+0000","last_active":"2024-03-04T15:41:14.788405+0000","last_peered":"2024-03-04T15:41:14.788405+0000","last_clean":"2024-03-04T15:41:14.788405+0000","last_became_active":"2024-03-04T15:40:47.739691+0000","last_became_peered":"2024-03-04T15:40:47.739691+0000","last_unstale":"2024-03-04T15:41:14.788405+0000","last_undegraded":"2024-03-04T15:41:14.788405+0000","last_fullsized":"2024-03-04T15:41:14.788405+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:18:11.713213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.790610+0000","last_change":"2024-03-04T15:40:51.769512+0000","last_active":"2024-03-04T15:41:14.790610+0000","last_peered":"2024-03-04T15:41:14.790610+0000","last_clean":"2024-03-04T15:41:14.790610+0000","last_became_active":"2024-03-04T15:40:51.768874+0000","last_became_peered":"2024-03-04T15:40:51.768874+0000","last_unstale":"2024-03-04T15:41:14.790610+0000","last_undegraded":"2024-03-04T15:41:14.790610+0000","last_fullsized":"2024-03-04T15:41:14.790610+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:32:36.165836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789162+0000","last_change":"2024-03-04T15:40:45.735494+0000","last_active":"2024-03-04T15:41:14.789162+0000","last_peered":"2024-03-04T15:41:14.789162+0000","last_clean":"2024-03-04T15:41:14.789162+0000","last_became_active":"2024-03-04T15:40:45.735155+0000","last_became_peered":"2024-03-04T15:40:45.735155+0000","last_unstale":"2024-03-04T15:41:14.789162+0000","last_undegraded":"2024-03-04T15:41:14.789162+0000","last_fullsized":"2024-03-04T15:41:14.789162+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:33:49.251366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"31'9","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785202+0000","last_change":"2024-03-04T15:40:47.739299+0000","last_active":"2024-03-04T15:41:14.785202+0000","last_peered":"2024-03-04T15:41:14.785202+0000","last_clean":"2024-03-04T15:41:14.785202+0000","last_became_active":"2024-03-04T15:40:47.738910+0000","last_became_peered":"2024-03-04T15:40:47.738910+0000","last_unstale":"2024-03-04T15:41:14.785202+0000","last_undegraded":"2024-03-04T15:41:14.785202+0000","last_fullsized":"2024-03-04T15:41:14.785202+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:57:22.219470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785908+0000","last_change":"2024-03-04T15:40:49.756320+0000","last_active":"2024-03-04T15:41:14.785908+0000","last_peered":"2024-03-04T15:41:14.785908+0000","last_clean":"2024-03-04T15:41:14.785908+0000","last_became_active":"2024-03-04T15:40:49.755969+0000","last_became_peered":"2024-03-04T15:40:49.755969+0000","last_unstale":"2024-03-04T15:41:14.785908+0000","last_undegraded":"2024-03-04T15:41:14.785908+0000","last_fullsized":"2024-03-04T15:41:14.785908+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:16:26.539391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787266+0000","last_change":"2024-03-04T15:40:51.769660+0000","last_active":"2024-03-04T15:41:14.787266+0000","last_peered":"2024-03-04T15:41:14.787266+0000","last_clean":"2024-03-04T15:41:14.787266+0000","last_became_active":"2024-03-04T15:40:51.768648+0000","last_became_peered":"2024-03-04T15:40:51.768648+0000","last_unstale":"2024-03-04T15:41:14.787266+0000","last_undegraded":"2024-03-04T15:41:14.787266+0000","last_fullsized":"2024-03-04T15:41:14.787266+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:20:55.287335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"31'8","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:30.783803+0000","last_change":"2024-03-04T15:40:49.754839+0000","last_active":"2024-03-04T15:41:30.783803+0000","last_peered":"2024-03-04T15:41:30.783803+0000","last_clean":"2024-03-04T15:41:30.783803+0000","last_became_active":"2024-03-04T15:40:49.752532+0000","last_became_peered":"2024-03-04T15:40:49.752532+0000","last_unstale":"2024-03-04T15:41:30.783803+0000","last_undegraded":"2024-03-04T15:41:30.783803+0000","last_fullsized":"2024-03-04T15:41:30.783803+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:46:59.866289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789051+0000","last_change":"2024-03-04T15:40:45.733961+0000","last_active":"2024-03-04T15:41:14.789051+0000","last_peered":"2024-03-04T15:41:14.789051+0000","last_clean":"2024-03-04T15:41:14.789051+0000","last_became_active":"2024-03-04T15:40:45.733529+0000","last_became_peered":"2024-03-04T15:40:45.733529+0000","last_unstale":"2024-03-04T15:41:14.789051+0000","last_undegraded":"2024-03-04T15:41:14.789051+0000","last_fullsized":"2024-03-04T15:41:14.789051+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:03:53.129668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"31'19","reported_seq":54,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785361+0000","last_change":"2024-03-04T15:40:47.742451+0000","last_active":"2024-03-04T15:41:14.785361+0000","last_peered":"2024-03-04T15:41:14.785361+0000","last_clean":"2024-03-04T15:41:14.785361+0000","last_became_active":"2024-03-04T15:40:47.741861+0000","last_became_peered":"2024-03-04T15:40:47.741861+0000","last_unstale":"2024-03-04T15:41:14.785361+0000","last_undegraded":"2024-03-04T15:41:14.785361+0000","last_fullsized":"2024-03-04T15:41:14.785361+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:29:21.105665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788488+0000","last_change":"2024-03-04T15:40:51.768299+0000","last_active":"2024-03-04T15:41:14.788488+0000","last_peered":"2024-03-04T15:41:14.788488+0000","last_clean":"2024-03-04T15:41:14.788488+0000","last_became_active":"2024-03-04T15:40:51.767982+0000","last_became_peered":"2024-03-04T15:40:51.767982+0000","last_unstale":"2024-03-04T15:41:14.788488+0000","last_undegraded":"2024-03-04T15:41:14.788488+0000","last_fullsized":"2024-03-04T15:41:14.788488+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:16:36.122393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.789795+0000","last_change":"2024-03-04T15:40:45.735836+0000","last_active":"2024-03-04T15:41:14.789795+0000","last_peered":"2024-03-04T15:41:14.789795+0000","last_clean":"2024-03-04T15:41:14.789795+0000","last_became_active":"2024-03-04T15:40:45.735362+0000","last_became_peered":"2024-03-04T15:40:45.735362+0000","last_unstale":"2024-03-04T15:41:14.789795+0000","last_undegraded":"2024-03-04T15:41:14.789795+0000","last_fullsized":"2024-03-04T15:41:14.789795+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:33.692098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"31'17","reported_seq":51,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787205+0000","last_change":"2024-03-04T15:40:47.742947+0000","last_active":"2024-03-04T15:41:14.787205+0000","last_peered":"2024-03-04T15:41:14.787205+0000","last_clean":"2024-03-04T15:41:14.787205+0000","last_became_active":"2024-03-04T15:40:47.742132+0000","last_became_peered":"2024-03-04T15:40:47.742132+0000","last_unstale":"2024-03-04T15:41:14.787205+0000","last_undegraded":"2024-03-04T15:41:14.787205+0000","last_fullsized":"2024-03-04T15:41:14.787205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:30:24.768011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784716+0000","last_change":"2024-03-04T15:40:49.756593+0000","last_active":"2024-03-04T15:41:14.784716+0000","last_peered":"2024-03-04T15:41:14.784716+0000","last_clean":"2024-03-04T15:41:14.784716+0000","last_became_active":"2024-03-04T15:40:49.752984+0000","last_became_peered":"2024-03-04T15:40:49.752984+0000","last_unstale":"2024-03-04T15:41:14.784716+0000","last_undegraded":"2024-03-04T15:41:14.784716+0000","last_fullsized":"2024-03-04T15:41:14.784716+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:36:10.914315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.783549+0000","last_change":"2024-03-04T15:40:51.765071+0000","last_active":"2024-03-04T15:41:14.783549+0000","last_peered":"2024-03-04T15:41:14.783549+0000","last_clean":"2024-03-04T15:41:14.783549+0000","last_became_active":"2024-03-04T15:40:51.763679+0000","last_became_peered":"2024-03-04T15:40:51.763679+0000","last_unstale":"2024-03-04T15:41:14.783549+0000","last_undegraded":"2024-03-04T15:41:14.783549+0000","last_fullsized":"2024-03-04T15:41:14.783549+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:41:07.548803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.790864+0000","last_change":"2024-03-04T15:40:45.733200+0000","last_active":"2024-03-04T15:41:14.790864+0000","last_peered":"2024-03-04T15:41:14.790864+0000","last_clean":"2024-03-04T15:41:14.790864+0000","last_became_active":"2024-03-04T15:40:45.732814+0000","last_became_peered":"2024-03-04T15:40:45.732814+0000","last_unstale":"2024-03-04T15:41:14.790864+0000","last_undegraded":"2024-03-04T15:41:14.790864+0000","last_fullsized":"2024-03-04T15:41:14.790864+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:38:52.744952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"31'10","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787222+0000","last_change":"2024-03-04T15:40:47.743481+0000","last_active":"2024-03-04T15:41:14.787222+0000","last_peered":"2024-03-04T15:41:14.787222+0000","last_clean":"2024-03-04T15:41:14.787222+0000","last_became_active":"2024-03-04T15:40:47.742836+0000","last_became_peered":"2024-03-04T15:40:47.742836+0000","last_unstale":"2024-03-04T15:41:14.787222+0000","last_undegraded":"2024-03-04T15:41:14.787222+0000","last_fullsized":"2024-03-04T15:41:14.787222+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:21.082802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784711+0000","last_change":"2024-03-04T15:40:49.759144+0000","last_active":"2024-03-04T15:41:14.784711+0000","last_peered":"2024-03-04T15:41:14.784711+0000","last_clean":"2024-03-04T15:41:14.784711+0000","last_became_active":"2024-03-04T15:40:49.758126+0000","last_became_peered":"2024-03-04T15:40:49.758126+0000","last_unstale":"2024-03-04T15:41:14.784711+0000","last_undegraded":"2024-03-04T15:41:14.784711+0000","last_fullsized":"2024-03-04T15:41:14.784711+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:42:44.534692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.784430+0000","last_change":"2024-03-04T15:40:51.762601+0000","last_active":"2024-03-04T15:41:14.784430+0000","last_peered":"2024-03-04T15:41:14.784430+0000","last_clean":"2024-03-04T15:41:14.784430+0000","last_became_active":"2024-03-04T15:40:51.762064+0000","last_became_peered":"2024-03-04T15:40:51.762064+0000","last_unstale":"2024-03-04T15:41:14.784430+0000","last_undegraded":"2024-03-04T15:41:14.784430+0000","last_fullsized":"2024-03-04T15:41:14.784430+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:50.749466+0000","last_clean_scrub_stamp":"2024-03-04T15:40:50.749466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:30:50.526073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.785920+0000","last_change":"2024-03-04T15:40:45.733534+0000","last_active":"2024-03-04T15:41:14.785920+0000","last_peered":"2024-03-04T15:41:14.785920+0000","last_clean":"2024-03-04T15:41:14.785920+0000","last_became_active":"2024-03-04T15:40:45.733162+0000","last_became_peered":"2024-03-04T15:40:45.733162+0000","last_unstale":"2024-03-04T15:41:14.785920+0000","last_undegraded":"2024-03-04T15:41:14.785920+0000","last_fullsized":"2024-03-04T15:41:14.785920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:44.707962+0000","last_clean_scrub_stamp":"2024-03-04T15:40:44.707962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:43:20.475611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"31'15","reported_seq":48,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.787019+0000","last_change":"2024-03-04T15:40:47.742033+0000","last_active":"2024-03-04T15:41:14.787019+0000","last_peered":"2024-03-04T15:41:14.787019+0000","last_clean":"2024-03-04T15:41:14.787019+0000","last_became_active":"2024-03-04T15:40:47.741705+0000","last_became_peered":"2024-03-04T15:40:47.741705+0000","last_unstale":"2024-03-04T15:41:14.787019+0000","last_undegraded":"2024-03-04T15:41:14.787019+0000","last_fullsized":"2024-03-04T15:41:14.787019+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:46.720479+0000","last_clean_scrub_stamp":"2024-03-04T15:40:46.720479+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:23:40.235430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-04T15:41:14.788630+0000","last_change":"2024-03-04T15:40:49.758940+0000","last_active":"2024-03-04T15:41:14.788630+0000","last_peered":"2024-03-04T15:41:14.788630+0000","last_clean":"2024-03-04T15:41:14.788630+0000","last_became_active":"2024-03-04T15:40:49.758640+0000","last_became_peered":"2024-03-04T15:40:49.758640+0000","last_unstale":"2024-03-04T15:41:14.788630+0000","last_undegraded":"2024-03-04T15:41:14.788630+0000","last_fullsized":"2024-03-04T15:41:14.788630+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:40:48.731231+0000","last_clean_scrub_stamp":"2024-03-04T15:40:48.731231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:47:41.637803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411344,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738395,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708844,"statfs":{"total":95990841344,"available":95957856256,"internally_reserved":0,"allocated":1060864,"data_stored":703168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T15:41:39.907 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-04T15:41:39.907 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-04T15:41:39.907 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-04T15:41:39.908 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- ceph health --format=json 2024-03-04T15:41:40.096 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:39 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:40.226 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:39 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:40.226 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:39 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:40.226 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:39 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:40.226 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:39 smithi080 ceph-mon[27564]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-04T15:41:40.226 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:39 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:40.226 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:39 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:40.226 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:39 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T15:41:40.226 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:39 smithi080 ceph-mon[27564]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T15:41:40.226 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:39 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:40.226 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:39 smithi080 ceph-mon[27564]: Deploying daemon grafana.a on smithi080 2024-03-04T15:41:40.227 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:39 smithi080 ceph-mon[27564]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:40.227 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:39 smithi080 ceph-mon[27564]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:41:40.729 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:41:40 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a[49242]: ts=2024-03-04T15:41:40.466Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000662954s 2024-03-04T15:41:40.756 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:40.994 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:40 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:40.994 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:40 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:41.237 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:40 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:41.238 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:40 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:41.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:40 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:41.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:40 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:42.124 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:41 smithi005 ceph-mon[25858]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:42.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:41 smithi087 ceph-mon[27664]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:42.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:41 smithi080 ceph-mon[27564]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:43.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:43 smithi005 ceph-mon[25858]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:44.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:43 smithi087 ceph-mon[27664]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:44.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:43 smithi080 ceph-mon[27564]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:44.706 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-04T15:41:44.706 INFO:teuthology.orchestra.run.smithi005.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-04T15:41:45.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:44 smithi087 ceph-mon[27664]: from='client.? 172.21.15.5:0/148832571' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T15:41:45.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:44 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:45.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:44 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:45.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:44 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:45.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:44 smithi005 ceph-mon[25858]: from='client.? 172.21.15.5:0/148832571' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T15:41:45.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:44 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:45.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:44 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:45.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:44 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:45.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:44 smithi080 ceph-mon[27564]: from='client.? 172.21.15.5:0/148832571' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T15:41:45.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:44 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:45.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:44 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:45.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:44 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:45.429 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-04T15:41:45.430 INFO:tasks.cephadm:Setup complete, yielding 2024-03-04T15:41:45.430 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T15:41:45.442 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi005.front.sepia.ceph.com 2024-03-04T15:41:45.442 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-04T15:41:45.700 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:46.027 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:45 smithi087 ceph-mon[27664]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:46.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:45 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:41:46.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:45 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:46.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:45 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:46.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:45 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:46.028 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:45 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:46.110 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:45 smithi005 ceph-mon[25858]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:46.110 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:45 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:41:46.110 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:45 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:46.111 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:45 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:46.111 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:45 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:46.111 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:45 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:46.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:45 smithi080 ceph-mon[27564]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:46.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:45 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:41:46.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:45 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:46.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:45 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:46.347 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:45 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:46.347 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:45 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:47.181 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:46 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: ::ffff:172.21.15.80 - - [04/Mar/2024:15:41:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T15:41:48.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:47 smithi005 ceph-mon[25858]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:48.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:47 smithi087 ceph-mon[27664]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:48.312 INFO:teuthology.orchestra.run.smithi005.stdout:167 167 2024-03-04T15:41:48.346 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:47 smithi080 ceph-mon[27564]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:48.681 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:41:48 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a[49242]: ts=2024-03-04T15:41:48.468Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002574121s 2024-03-04T15:41:48.838 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- bash -c 'ceph orch status' 2024-03-04T15:41:49.084 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:49.518 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:49 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:49.519 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:49 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:49.519 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:49 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:49.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:49 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:49.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:49 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:49.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:49 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:49.723 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:49 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:49.723 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:49 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:49.723 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:49 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:50.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:50 smithi080 ceph-mon[27564]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:50.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:50 smithi005 ceph-mon[25858]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout:Backend: cephadm 2024-03-04T15:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout:Available: Yes 2024-03-04T15:41:50.726 INFO:teuthology.orchestra.run.smithi005.stdout:Paused: No 2024-03-04T15:41:50.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:50 smithi087 ceph-mon[27664]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:51.339 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-04T15:41:51.583 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:52.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:52 smithi080 ceph-mon[27564]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:41:52.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:52 smithi080 ceph-mon[27564]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:52.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:52 smithi005 ceph-mon[25858]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:41:52.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:52 smithi005 ceph-mon[25858]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:52.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:52 smithi087 ceph-mon[27664]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:41:52.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:52 smithi087 ceph-mon[27664]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:53.480 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T15:41:53.480 INFO:teuthology.orchestra.run.smithi005.stdout:agent.smithi005 smithi005 running 5s ago 3m - - 2024-03-04T15:41:53.480 INFO:teuthology.orchestra.run.smithi005.stdout:agent.smithi080 smithi080 running 8s ago 3m - - 2024-03-04T15:41:53.480 INFO:teuthology.orchestra.run.smithi005.stdout:agent.smithi087 smithi087 running 7s ago 3m - - 2024-03-04T15:41:53.480 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (15s) 5s ago 14s 16.5M - 0.25.0 c8568f914cd2 f2f4e735e3ca 2024-03-04T15:41:53.480 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi080 *:8443,8765 running (2m) 8s ago 2m 448M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e b03f37862121 2024-03-04T15:41:53.480 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:9283,8765 running (4m) 5s ago 4m 502M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e f139651d923a 2024-03-04T15:41:53.481 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.z smithi087 *:8443,8765 running (2m) 7s ago 2m 448M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4f153c08506c 2024-03-04T15:41:53.481 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (4m) 5s ago 4m 47.9M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e c077183e60f8 2024-03-04T15:41:53.481 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi080 running (3m) 8s ago 3m 37.8M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 767a142e0f99 2024-03-04T15:41:53.481 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi087 running (3m) 7s ago 3m 37.4M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 22502b7796bd 2024-03-04T15:41:53.481 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (2m) 5s ago 2m 61.2M 4096M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e de20379aebe7 2024-03-04T15:41:53.481 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi080 running (117s) 8s ago 117s 59.0M 4096M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 6998c7ee4d77 2024-03-04T15:41:53.482 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi087 running (91s) 7s ago 90s 51.9M 4096M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 03f1dd7f9a87 2024-03-04T15:41:53.482 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi080 *:9095 running (47s) 8s ago 47s 29.3M - 2.43.0 a07b618ecd1d 713fb4e36d74 2024-03-04T15:41:53.482 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.a smithi005 *:80 running (69s) 5s ago 69s 76.4M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 3c0a150b7041 2024-03-04T15:41:54.068 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-04T15:41:54.309 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:54.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:54 smithi005 ceph-mon[25858]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:54.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:54 smithi005 ceph-mon[25858]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:41:54.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:54 smithi080 ceph-mon[27564]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:54.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:54 smithi080 ceph-mon[27564]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:41:54.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:54 smithi087 ceph-mon[27664]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:54.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:54 smithi087 ceph-mon[27664]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:41:56.076 INFO:teuthology.orchestra.run.smithi005.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-04T15:41:56.077 INFO:teuthology.orchestra.run.smithi005.stdout:agent 3/3 11s ago 41s * 2024-03-04T15:41:56.077 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager ?:9093,9094 1/1 7s ago 66s smithi005=a;count:1 2024-03-04T15:41:56.077 INFO:teuthology.orchestra.run.smithi005.stdout:grafana ?:3000 0/1 - 63s smithi080=a;count:1 2024-03-04T15:41:56.077 INFO:teuthology.orchestra.run.smithi005.stdout:mgr 3/3 11s ago 2m smithi005=y;smithi080=x;smithi087=z;count:3 2024-03-04T15:41:56.077 INFO:teuthology.orchestra.run.smithi005.stdout:mon 3/3 11s ago 3m smithi005:172.21.15.5=a;smithi080:172.21.15.80=b;smithi087:172.21.15.87=c;count:3 2024-03-04T15:41:56.077 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter ?:9100 0/3 - 69s smithi005=a;smithi080=b;smithi087=c;count:3 2024-03-04T15:41:56.077 INFO:teuthology.orchestra.run.smithi005.stdout:osd 3 11s ago - 2024-03-04T15:41:56.077 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus ?:9095 1/1 11s ago 71s smithi080=a;count:1 2024-03-04T15:41:56.077 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.a ?:80 1/1 7s ago 71s smithi005=foo.a;count:1 2024-03-04T15:41:56.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:56 smithi005 ceph-mon[25858]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:56.497 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:56 smithi080 ceph-mon[27564]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:56.697 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-04T15:41:56.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:56 smithi087 ceph-mon[27664]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:56.846 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:56 smithi080 systemd[1]: Starting Ceph grafana.a for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:41:57.042 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:41:56 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: ::ffff:172.21.15.80 - - [04/Mar/2024:15:41:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T15:41:57.148 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:56 smithi080 podman[38842]: 2024-03-04 15:41:56.795426022 +0000 UTC m=+0.016655420 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-04T15:41:57.148 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:56 smithi080 podman[38842]: 2024-03-04 15:41:56.897696285 +0000 UTC m=+0.118925673 container create 9508aed1abeea54fbdcc1ae0478c554891062361a013c03d54493be27328330c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=239.1651231664, description=Ceph Grafana Container, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., name=ubi8, distribution-scope=public, architecture=x86_64, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.buildah.version=1.32.0, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-04T15:41:57.148 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 podman[38842]: 2024-03-04 15:41:57.14165184 +0000 UTC m=+0.362881226 container init 9508aed1abeea54fbdcc1ae0478c554891062361a013c03d54493be27328330c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, io.buildah.version=1.32.0, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, version=8.5, release=239.1651231664) 2024-03-04T15:41:57.148 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 podman[38842]: 2024-03-04 15:41:57.14696539 +0000 UTC m=+0.368194778 container start 9508aed1abeea54fbdcc1ae0478c554891062361a013c03d54493be27328330c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, build-date=2022-04-29T11:48:11.055897, name=ubi8, distribution-scope=public, architecture=x86_64, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.component=ubi8-container, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-04T15:41:57.400 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-mon[27564]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:41:57.402 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 bash[38842]: 9508aed1abeea54fbdcc1ae0478c554891062361a013c03d54493be27328330c 2024-03-04T15:41:57.402 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 systemd[1]: Started Ceph grafana.a for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:41:57.402 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=settings t=2024-03-04T15:41:57.295620825Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-04T15:41:57.402 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=settings t=2024-03-04T15:41:57.29582994Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-04T15:41:57.403 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=settings t=2024-03-04T15:41:57.295848325Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-04T15:41:57.403 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=settings t=2024-03-04T15:41:57.295855715Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-04T15:41:57.403 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=settings t=2024-03-04T15:41:57.295861798Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-04T15:41:57.403 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=settings t=2024-03-04T15:41:57.295867599Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-04T15:41:57.403 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=settings t=2024-03-04T15:41:57.295873845Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-04T15:41:57.403 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=settings t=2024-03-04T15:41:57.295885685Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-04T15:41:57.404 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=settings t=2024-03-04T15:41:57.295892916Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-04T15:41:57.404 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=settings t=2024-03-04T15:41:57.295898678Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-04T15:41:57.404 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=settings t=2024-03-04T15:41:57.295903976Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-04T15:41:57.404 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=settings t=2024-03-04T15:41:57.295909556Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-04T15:41:57.404 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=settings t=2024-03-04T15:41:57.295915269Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-04T15:41:57.404 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=settings t=2024-03-04T15:41:57.29592632Z level=info msg="App mode production" 2024-03-04T15:41:57.405 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=sqlstore t=2024-03-04T15:41:57.295971975Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-04T15:41:57.405 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=sqlstore t=2024-03-04T15:41:57.295987694Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-04T15:41:57.405 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.296832229Z level=info msg="Starting DB migrations" 2024-03-04T15:41:57.405 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.296915819Z level=info msg="Executing migration" id="create migration_log table" 2024-03-04T15:41:57.405 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.297458085Z level=info msg="Executing migration" id="create user table" 2024-03-04T15:41:57.406 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.297916692Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-04T15:41:57.406 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.298499144Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-04T15:41:57.406 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.298919922Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-04T15:41:57.406 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.299350055Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-04T15:41:57.406 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.299697701Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-04T15:41:57.406 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.300410416Z level=info msg="Executing migration" id="create user table v2" 2024-03-04T15:41:57.407 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.300791569Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-04T15:41:57.407 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.301186374Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-04T15:41:57.407 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.301602526Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-04T15:41:57.407 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.301914392Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-04T15:41:57.407 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.302258409Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-04T15:41:57.407 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.302637808Z level=info msg="Executing migration" id="Update user table charset" 2024-03-04T15:41:57.408 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.302916769Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-04T15:41:57.408 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.303380696Z level=info msg="Executing migration" id="Add missing user data" 2024-03-04T15:41:57.408 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.303704395Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-04T15:41:57.408 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.304846831Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-04T15:41:57.408 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.305280026Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-04T15:41:57.408 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.305680673Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-04T15:41:57.408 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.307327044Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-04T15:41:57.409 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.307805963Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-04T15:41:57.409 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.308440855Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-04T15:41:57.409 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.30882764Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-04T15:41:57.409 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.309227104Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-04T15:41:57.409 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.309644725Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-04T15:41:57.409 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.309918145Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-04T15:41:57.410 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.310364211Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-04T15:41:57.410 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.310720668Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-04T15:41:57.410 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.311115968Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-04T15:41:57.410 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.311509291Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-04T15:41:57.410 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.312152292Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-04T15:41:57.410 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.312591844Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-04T15:41:57.410 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.312988697Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-04T15:41:57.411 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.313392251Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-04T15:41:57.411 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.313738239Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-04T15:41:57.411 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.314151392Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-04T15:41:57.411 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.31448404Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-04T15:41:57.411 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.314790763Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-04T15:41:57.411 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.31518695Z level=info msg="Executing migration" id="create star table" 2024-03-04T15:41:57.412 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.315552258Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-04T15:41:57.412 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.315962213Z level=info msg="Executing migration" id="create org table v1" 2024-03-04T15:41:57.412 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.317227827Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-04T15:41:57.412 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.317646726Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-04T15:41:57.412 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.318469077Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-04T15:41:57.412 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.319288641Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-04T15:41:57.413 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.320278099Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-04T15:41:57.413 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.320969331Z level=info msg="Executing migration" id="Update org table charset" 2024-03-04T15:41:57.413 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.321937691Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-04T15:41:57.413 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.322593821Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-04T15:41:57.413 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.32340698Z level=info msg="Executing migration" id="create dashboard table" 2024-03-04T15:41:57.413 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.323825307Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-04T15:41:57.414 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.324288006Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-04T15:41:57.414 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.325114182Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-04T15:41:57.414 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.325904526Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-04T15:41:57.414 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.326750138Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-04T15:41:57.414 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.327621848Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-04T15:41:57.414 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.328506424Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-04T15:41:57.415 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.329286676Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-04T15:41:57.415 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.330084397Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-04T15:41:57.415 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.330994468Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-04T15:41:57.415 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.33184855Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-04T15:41:57.415 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.332677319Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-04T15:41:57.415 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.333488869Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-04T15:41:57.415 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.33398038Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-04T15:41:57.416 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.334460905Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-04T15:41:57.416 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.335286505Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-04T15:41:57.416 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.335728895Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-04T15:41:57.416 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.336253022Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-04T15:41:57.416 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.337195883Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-04T15:41:57.416 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.337637283Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-04T15:41:57.416 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.338057161Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-04T15:41:57.417 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.338353394Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-04T15:41:57.417 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.338982606Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-04T15:41:57.417 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.339811218Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-04T15:41:57.417 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.340676477Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-04T15:41:57.417 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.341522144Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-04T15:41:57.417 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.342380796Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-04T15:41:57.419 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.345355079Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-04T15:41:57.419 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.348185967Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-04T15:41:57.419 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.348560458Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-04T15:41:57.419 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.351281918Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-04T15:41:57.419 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.351698282Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-04T15:41:57.419 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.354041091Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-04T15:41:57.420 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.356965306Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-04T15:41:57.420 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.35757428Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-04T15:41:57.420 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.360574485Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-04T15:41:57.420 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.363843709Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-04T15:41:57.420 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.366750476Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-04T15:41:57.420 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.367273949Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-04T15:41:57.420 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.370095029Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-04T15:41:57.421 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.372854337Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-04T15:41:57.421 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.375661402Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-04T15:41:57.421 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.376119186Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-04T15:41:57.421 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.378934976Z level=info msg="Executing migration" id="create data_source table" 2024-03-04T15:41:57.421 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.38171075Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-04T15:41:57.421 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.384494844Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-04T15:41:57.421 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.384968673Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-04T15:41:57.421 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.387596776Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-04T15:41:57.421 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.390313782Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-04T15:41:57.421 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.392989108Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-04T15:41:57.422 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.393544022Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-04T15:41:57.422 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.396159417Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-04T15:41:57.422 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.396725165Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-04T15:41:57.422 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.398964552Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-04T15:41:57.422 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.401566989Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-04T15:41:57.481 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:41:57.602 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:57 smithi005 ceph-mon[25858]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:41:57.652 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.402354383Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-04T15:41:57.652 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.405132033Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-04T15:41:57.652 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.407837226Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-04T15:41:57.652 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.410570425Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-04T15:41:57.652 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.410914394Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-04T15:41:57.652 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.413712114Z level=info msg="Executing migration" id="Add uid column" 2024-03-04T15:41:57.652 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.416240323Z level=info msg="Executing migration" id="Update uid value" 2024-03-04T15:41:57.653 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.417175825Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-04T15:41:57.653 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.417714016Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-04T15:41:57.653 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.418640165Z level=info msg="Executing migration" id="create api_key table" 2024-03-04T15:41:57.653 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.419329712Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-04T15:41:57.653 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.420214826Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-04T15:41:57.653 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.42079111Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-04T15:41:57.653 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.42161483Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-04T15:41:57.654 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.422498798Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-04T15:41:57.654 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.423141721Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-04T15:41:57.654 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.424133434Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-04T15:41:57.654 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.425244366Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-04T15:41:57.654 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.426173606Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-04T15:41:57.654 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.426888266Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-04T15:41:57.654 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.427838301Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-04T15:41:57.655 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.42860773Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-04T15:41:57.655 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.429456428Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-04T15:41:57.655 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.429922144Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-04T15:41:57.655 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.430221751Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-04T15:41:57.655 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.431176598Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-04T15:41:57.655 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.431923242Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-04T15:41:57.656 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.432230742Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-04T15:41:57.656 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.433065894Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-04T15:41:57.656 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.433605198Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-04T15:41:57.656 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.433962037Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-04T15:41:57.656 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.434327611Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-04T15:41:57.656 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.434784553Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-04T15:41:57.656 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.435212208Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-04T15:41:57.657 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.435657765Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-04T15:41:57.657 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.436617456Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-04T15:41:57.657 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.437363617Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-04T15:41:57.657 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.43834947Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-04T15:41:57.657 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.439126094Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-04T15:41:57.657 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.439998449Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-04T15:41:57.657 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.440982499Z level=info msg="Executing migration" id="create quota table v1" 2024-03-04T15:41:57.657 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.44168846Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-04T15:41:57.657 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.444756414Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-04T15:41:57.657 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.448365489Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-04T15:41:57.657 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.451008941Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-04T15:41:57.657 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.453889395Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-04T15:41:57.657 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.456364334Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-04T15:41:57.658 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.45934952Z level=info msg="Executing migration" id="create session table" 2024-03-04T15:41:57.658 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.462179515Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-04T15:41:57.658 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.464845836Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-04T15:41:57.658 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.467573639Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-04T15:41:57.658 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.4711827Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-04T15:41:57.658 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.473967388Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-04T15:41:57.658 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.476724082Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-04T15:41:57.658 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.479616649Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-04T15:41:57.658 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.482392665Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-04T15:41:57.658 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.485182187Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-04T15:41:57.658 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.488088882Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-04T15:41:57.660 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.490696416Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-04T15:41:57.660 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.493361339Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-04T15:41:57.660 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.496290143Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-04T15:41:57.660 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.49893239Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-04T15:41:57.660 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.501527498Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-04T15:41:57.660 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.504246022Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-04T15:41:57.660 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.506908275Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-04T15:41:57.660 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.509605907Z level=info msg="Executing migration" id="create alert table v1" 2024-03-04T15:41:57.660 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.512222057Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-04T15:41:57.660 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.514903108Z level=info msg="Executing migration" id="add index alert state" 2024-03-04T15:41:57.660 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.51753265Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-04T15:41:57.660 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.518001916Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-04T15:41:57.661 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.518421735Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-04T15:41:57.661 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.518877851Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-04T15:41:57.661 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.519323381Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-04T15:41:57.661 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.520749371Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-04T15:41:57.661 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.521188758Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-04T15:41:57.661 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.521651528Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-04T15:41:57.661 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.521966731Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-04T15:41:57.661 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.522344395Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-04T15:41:57.661 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.522751927Z level=info msg="Executing migration" id="Add column is_default" 2024-03-04T15:41:57.661 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.523443303Z level=info msg="Executing migration" id="Add column frequency" 2024-03-04T15:41:57.661 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.524117963Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-04T15:41:57.661 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.524789583Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-04T15:41:57.661 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.525493349Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-04T15:41:57.661 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.525960321Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-04T15:41:57.662 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.526306418Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-04T15:41:57.662 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.52661301Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-04T15:41:57.662 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.527049421Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-04T15:41:57.662 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.527539802Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-04T15:41:57.662 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.527920109Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-04T15:41:57.662 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.528361229Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-04T15:41:57.662 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.528766313Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-04T15:41:57.662 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.52943496Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-04T15:41:57.662 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.530130303Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-04T15:41:57.662 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.530496094Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-04T15:41:57.662 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.530961569Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-04T15:41:57.662 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.531400992Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-04T15:41:57.662 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.532084915Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-04T15:41:57.662 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.532433884Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-04T15:41:57.663 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.532848059Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-04T15:41:57.663 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.533308166Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-04T15:41:57.663 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.533622186Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-04T15:41:57.663 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.534115255Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-04T15:41:57.663 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.536931882Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-04T15:41:57.663 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.537402142Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-04T15:41:57.663 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.538329246Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-04T15:41:57.663 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.538794834Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-04T15:41:57.663 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.541777128Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-04T15:41:57.664 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.542184074Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-04T15:41:57.664 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.543091546Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-04T15:41:57.664 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.543517978Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-04T15:41:57.664 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.544211072Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-04T15:41:57.664 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.544634925Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-04T15:41:57.664 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.545156011Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-04T15:41:57.665 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.545610002Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-04T15:41:57.665 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.547163353Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-04T15:41:57.665 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.547583191Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-04T15:41:57.665 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.548060947Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-04T15:41:57.665 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.548391845Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-04T15:41:57.665 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.548740617Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-04T15:41:57.665 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.549076631Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-04T15:41:57.666 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.549773522Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-04T15:41:57.666 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.550480856Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-04T15:41:57.666 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.550927136Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-04T15:41:57.666 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.551365806Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-04T15:41:57.666 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.55167173Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-04T15:41:57.666 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.552401822Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-04T15:41:57.666 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.553131258Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-04T15:41:57.667 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.553483198Z level=info msg="Executing migration" id="Move region to single row" 2024-03-04T15:41:57.667 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.553841556Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-04T15:41:57.667 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.554271204Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-04T15:41:57.667 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.554707873Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-04T15:41:57.667 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.555150425Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-04T15:41:57.667 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.555579087Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-04T15:41:57.667 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.555967462Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-04T15:41:57.668 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.556393141Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-04T15:41:57.668 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.556688065Z level=info msg="Executing migration" id="create test_data table" 2024-03-04T15:41:57.668 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.557156135Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-04T15:41:57.668 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.557561491Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-04T15:41:57.668 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.557984325Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-04T15:41:57.668 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.558413992Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-04T15:41:57.668 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.558716072Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-04T15:41:57.669 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.559070303Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-04T15:41:57.669 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.559387171Z level=info msg="Executing migration" id="create team table" 2024-03-04T15:41:57.669 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.559777205Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-04T15:41:57.669 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.560241863Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-04T15:41:57.669 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.5637008Z level=info msg="Executing migration" id="create team member table" 2024-03-04T15:41:57.669 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.566014071Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-04T15:41:57.669 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.566496553Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-04T15:41:57.669 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.566983326Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-04T15:41:57.670 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.567430719Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-04T15:41:57.670 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.568186285Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-04T15:41:57.670 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.568939552Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-04T15:41:57.670 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.570648009Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-04T15:41:57.670 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.571111885Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-04T15:41:57.670 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.571603858Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-04T15:41:57.670 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.574356353Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-04T15:41:57.671 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.574803179Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-04T15:41:57.671 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.577740173Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-04T15:41:57.671 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.580630218Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-04T15:41:57.671 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.58353243Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-04T15:41:57.671 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.586446466Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-04T15:41:57.671 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.589130011Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-04T15:41:57.671 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.589478346Z level=info msg="Executing migration" id="create tag table" 2024-03-04T15:41:57.672 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.592257259Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-04T15:41:57.672 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.594670309Z level=info msg="Executing migration" id="create login attempt table" 2024-03-04T15:41:57.672 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.597378923Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-04T15:41:57.672 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.597983609Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-04T15:41:57.672 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.600690104Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-04T15:41:57.672 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.603193299Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-04T15:41:57.672 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.605932869Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-04T15:41:57.673 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.60637731Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-04T15:41:57.673 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.608946145Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-04T15:41:57.673 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.609339931Z level=info msg="Executing migration" id="create user auth table" 2024-03-04T15:41:57.673 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.61160451Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-04T15:41:57.673 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.612075526Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-04T15:41:57.673 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.614296895Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-04T15:41:57.673 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.615085915Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-04T15:41:57.674 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.617960226Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-04T15:41:57.674 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.620686957Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-04T15:41:57.674 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.623389144Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-04T15:41:57.674 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.623911852Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-04T15:41:57.674 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.624681175Z level=info msg="Executing migration" id="create server_lock table" 2024-03-04T15:41:57.674 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.625527129Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-04T15:41:57.675 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.626543766Z level=info msg="Executing migration" id="create user auth token table" 2024-03-04T15:41:57.675 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.627082446Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-04T15:41:57.675 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.627907929Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-04T15:41:57.675 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.628728187Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-04T15:41:57.675 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.629573257Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-04T15:41:57.675 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.630717339Z level=info msg="Executing migration" id="create cache_data table" 2024-03-04T15:41:57.675 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.631572171Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-04T15:41:57.676 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.632324238Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-04T15:41:57.676 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.633162749Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-04T15:41:57.676 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.633602626Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-04T15:41:57.676 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.634423943Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-04T15:41:57.676 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.634745091Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-04T15:41:57.676 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.635263625Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-04T15:41:57.677 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.636114902Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-04T15:41:57.677 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.636974949Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-04T15:41:57.677 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.637806515Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-04T15:41:57.677 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.638694965Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-04T15:41:57.677 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.639542727Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-04T15:41:57.677 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.640397003Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-04T15:41:57.677 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.641259263Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-04T15:41:57.677 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.642093505Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-04T15:41:57.678 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.643019307Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-04T15:41:57.678 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.643888991Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-04T15:41:57.678 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.644718675Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-04T15:41:57.678 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.645568362Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-04T15:41:57.679 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.646381068Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-04T15:41:57.679 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.647234148Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-04T15:41:57.679 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.648091374Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-04T15:41:57.679 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.648979836Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-04T15:41:57.679 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.649814206Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-04T15:41:57.679 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.652279745Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-04T15:41:57.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:57 smithi087 ceph-mon[27664]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:41:57.901 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.654918859Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-04T15:41:57.902 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.657654355Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-04T15:41:57.902 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.660237241Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-04T15:41:57.902 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.664908651Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-04T15:41:57.902 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.66838406Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-04T15:41:57.902 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.669203857Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-04T15:41:57.902 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.670118993Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-04T15:41:57.902 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.670980055Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-04T15:41:57.902 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.671962371Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-04T15:41:57.902 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.672630726Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-04T15:41:57.903 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.673417246Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-04T15:41:57.903 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.674315446Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-04T15:41:57.903 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.67506387Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-04T15:41:57.903 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.675872508Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-04T15:41:57.903 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.677072954Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-04T15:41:57.903 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.678306538Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-04T15:41:57.903 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.679894067Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-04T15:41:57.903 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.680571602Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-04T15:41:57.904 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.681766853Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-04T15:41:57.904 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.68261037Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-04T15:41:57.904 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.683499587Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-04T15:41:57.904 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.684357544Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-04T15:41:57.904 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.685177237Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-04T15:41:57.904 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.686109095Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-04T15:41:57.904 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.687084995Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-04T15:41:57.904 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.687921657Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-04T15:41:57.904 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.688605269Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-04T15:41:57.904 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.689509268Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-04T15:41:57.904 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.690545895Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-04T15:41:57.904 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.691417361Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-04T15:41:57.904 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.692292865Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-04T15:41:57.904 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.693139448Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-04T15:41:57.905 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.693995663Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-04T15:41:57.905 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.694845321Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-04T15:41:57.905 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.695698319Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-04T15:41:57.906 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.696571176Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-04T15:41:57.906 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.697506178Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-04T15:41:57.906 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.698455071Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-04T15:41:57.906 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.701529939Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-04T15:41:57.906 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.704105641Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-04T15:41:57.906 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.706847176Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-04T15:41:57.906 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.709722343Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-04T15:41:57.906 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.712631059Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-04T15:41:57.906 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.71561988Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-04T15:41:57.907 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.7183838Z level=info msg="Executing migration" id="create alert_image table" 2024-03-04T15:41:57.907 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.721600751Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-04T15:41:57.907 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.723719035Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-04T15:41:57.907 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.726664547Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-04T15:41:57.907 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.729515669Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-04T15:41:57.907 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.732180694Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-04T15:41:57.907 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.732269956Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-04T15:41:57.907 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.734983979Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-04T15:41:57.907 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.737844442Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-04T15:41:57.907 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.74078586Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-04T15:41:57.907 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.74092831Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-04T15:41:57.907 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.741059674Z level=info msg="alerts found to migrate" alerts=0 2024-03-04T15:41:57.907 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.743556411Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-04T15:41:57.907 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.746202222Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-04T15:41:57.907 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.749103063Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-04T15:41:57.908 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.751602018Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-04T15:41:57.908 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.754373721Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-04T15:41:57.908 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.757012753Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-04T15:41:57.908 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.759518752Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-04T15:41:57.908 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.762254838Z level=info msg="Executing migration" id="create data_keys table" 2024-03-04T15:41:57.908 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.76506936Z level=info msg="Executing migration" id="create secrets table" 2024-03-04T15:41:57.908 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.767550633Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-04T15:41:57.908 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.77184287Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-04T15:41:57.908 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.774413267Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-04T15:41:57.908 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.777303199Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-04T15:41:57.908 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.781790295Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-04T15:41:57.908 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.785996933Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-04T15:41:57.908 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.788829997Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-04T15:41:57.909 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.791349688Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-04T15:41:57.909 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.79393207Z level=info msg="Executing migration" id="create permission table" 2024-03-04T15:41:57.909 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.796618797Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-04T15:41:57.909 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.799419641Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-04T15:41:57.909 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.80020698Z level=info msg="Executing migration" id="create role table" 2024-03-04T15:41:57.909 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.801009846Z level=info msg="Executing migration" id="add column display_name" 2024-03-04T15:41:57.909 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.801940604Z level=info msg="Executing migration" id="add column group_name" 2024-03-04T15:41:57.909 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.802855238Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-04T15:41:57.909 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.803706344Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-04T15:41:57.909 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.804561503Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-04T15:41:57.909 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.805382013Z level=info msg="Executing migration" id="create team role table" 2024-03-04T15:41:57.909 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.806279853Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-04T15:41:57.910 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.806959982Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-04T15:41:57.910 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.807942141Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-04T15:41:57.910 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.808654725Z level=info msg="Executing migration" id="create user role table" 2024-03-04T15:41:57.910 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.809532295Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-04T15:41:57.910 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.810238673Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-04T15:41:57.910 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.811112685Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-04T15:41:57.910 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.812096038Z level=info msg="Executing migration" id="create builtin role table" 2024-03-04T15:41:57.910 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.812780303Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-04T15:41:57.910 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.813692669Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-04T15:41:57.910 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.814457008Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-04T15:41:57.910 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.815453602Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-04T15:41:57.910 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.816302694Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-04T15:41:57.910 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.817149657Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-04T15:41:57.911 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.818005276Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-04T15:41:57.911 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.818914066Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-04T15:41:57.911 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.819743749Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-04T15:41:57.911 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.820688655Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-04T15:41:57.911 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.821722444Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-04T15:41:57.911 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.822560367Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-04T15:41:57.911 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.823445187Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-04T15:41:57.911 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.824296691Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-04T15:41:57.911 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.825151066Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-04T15:41:57.911 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.826058308Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-04T15:41:57.911 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.826914415Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-04T15:41:57.911 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.82779047Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-04T15:41:57.911 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.828653509Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-04T15:41:57.911 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.829505953Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-04T15:41:57.912 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.832471835Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-04T15:41:57.912 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.835319011Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-04T15:41:57.912 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.838038359Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-04T15:41:57.912 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.840823983Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-04T15:41:57.912 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.843607814Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-04T15:41:57.912 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.846527924Z level=info msg="Executing migration" id="add correlation config column" 2024-03-04T15:41:57.912 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.849306831Z level=info msg="Executing migration" id="create entity_events table" 2024-03-04T15:41:57.912 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.852082134Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-04T15:41:57.912 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.85492535Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-04T15:41:57.912 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.855020463Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-04T15:41:57.912 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.857916539Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-04T15:41:57.912 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.858024945Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-04T15:41:57.912 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.860570571Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-04T15:41:57.913 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.863343512Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-04T15:41:57.913 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.866116356Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-04T15:41:57.913 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.868980725Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-04T15:41:57.913 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.871927934Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-04T15:41:57.913 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.874657257Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-04T15:41:57.913 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.877278026Z level=info msg="Executing migration" id="Drop public config table" 2024-03-04T15:41:57.913 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.880049058Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-04T15:41:57.914 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.882679936Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-04T15:41:57.914 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.88529402Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-04T15:41:57.914 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.885763427Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-04T15:41:57.914 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.888479619Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-04T15:41:57.914 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.89150866Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-04T15:41:57.914 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.894560736Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-04T15:41:57.915 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.897271963Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-04T15:41:57.915 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.900024003Z level=info msg="Executing migration" id="add share column" 2024-03-04T15:41:57.915 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.901624951Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-04T15:41:57.915 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.902181007Z level=info msg="Executing migration" id="create file table" 2024-03-04T15:41:58.218 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.90447581Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-04T15:41:58.218 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.907259995Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-04T15:41:58.218 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.910049764Z level=info msg="Executing migration" id="create file_meta table" 2024-03-04T15:41:58.218 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.910433895Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-04T15:41:58.218 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.910868516Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-04T15:41:58.219 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.911177014Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-04T15:41:58.219 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.911533675Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-04T15:41:58.219 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.911826898Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-04T15:41:58.219 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.912280842Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-04T15:41:58.219 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.913308995Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-04T15:41:58.219 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.913624171Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-04T15:41:58.220 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.914075694Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-04T15:41:58.220 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.914467676Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-04T15:41:58.220 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.914763127Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-04T15:41:58.220 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.915143503Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-04T15:41:58.220 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.916175723Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-04T15:41:58.220 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.91960945Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-04T15:41:58.221 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.922420492Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-04T15:41:58.221 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.935850772Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-04T15:41:58.221 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.940072058Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-04T15:41:58.222 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.940538393Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-04T15:41:58.222 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.943979755Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-04T15:41:58.222 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.946664436Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-04T15:41:58.222 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.947032803Z level=info msg="Executing migration" id="create folder table" 2024-03-04T15:41:58.222 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.947445733Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-04T15:41:58.222 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.947990739Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-04T15:41:58.223 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.948434532Z level=info msg="Executing migration" id="Update folder title length" 2024-03-04T15:41:58.223 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.948689633Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-04T15:41:58.223 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=migrator t=2024-03-04T15:41:57.94911496Z level=info msg="migrations completed" performed=485 skipped=0 duration=652.223723ms 2024-03-04T15:41:58.223 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:57 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=sqlstore t=2024-03-04T15:41:57.94949532Z level=info msg="Created default organization" 2024-03-04T15:41:58.223 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=plugin.loader t=2024-03-04T15:41:58.005981216Z level=info msg="Plugin registered" pluginID=input 2024-03-04T15:41:58.223 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=plugin.loader t=2024-03-04T15:41:58.034478275Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-04T15:41:58.224 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=plugin.loader t=2024-03-04T15:41:58.034508005Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-04T15:41:58.224 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=secrets t=2024-03-04T15:41:58.034546979Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-04T15:41:58.224 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=query_data t=2024-03-04T15:41:58.036705962Z level=info msg="Query Service initialization" 2024-03-04T15:41:58.224 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=live.push_http t=2024-03-04T15:41:58.03925841Z level=info msg="Live Push Gateway initialization" 2024-03-04T15:41:58.481 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-mon[27564]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:58.481 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:58.481 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:58.482 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:58.482 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:58.482 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-mon[27564]: Deploying daemon node-exporter.a on smithi005 2024-03-04T15:41:58.530 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:58 smithi005 ceph-mon[25858]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:58.530 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:58 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:58.530 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:58 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:58.530 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:58 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:58.530 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:58 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:58.531 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:41:58 smithi005 ceph-mon[25858]: Deploying daemon node-exporter.a on smithi005 2024-03-04T15:41:58.736 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=infra.usagestats.collector t=2024-03-04T15:41:58.715701202Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-04T15:41:58.736 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=provisioning.datasources t=2024-03-04T15:41:58.716735141Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-04T15:41:58.736 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=provisioning.datasources t=2024-03-04T15:41:58.727997733Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-04T15:41:58.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:58 smithi087 ceph-mon[27664]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:41:58.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:58 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:58.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:58 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:58.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:58 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:58.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:58 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:41:58.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:41:58 smithi087 ceph-mon[27664]: Deploying daemon node-exporter.a on smithi005 2024-03-04T15:41:59.037 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=provisioning.alerting t=2024-03-04T15:41:58.739378648Z level=info msg="starting to provision alerting" 2024-03-04T15:41:59.037 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=provisioning.alerting t=2024-03-04T15:41:58.739401968Z level=info msg="finished to provision alerting" 2024-03-04T15:41:59.037 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=ngalert.state.manager t=2024-03-04T15:41:58.739565524Z level=info msg="Warming state cache for startup" 2024-03-04T15:41:59.037 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=grafanaStorageLogger t=2024-03-04T15:41:58.740359276Z level=info msg="storage starting" 2024-03-04T15:41:59.037 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=http.server t=2024-03-04T15:41:58.741870784Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-04T15:41:59.037 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=ngalert.state.manager t=2024-03-04T15:41:58.746755586Z level=info msg="State cache has been initialized" states=0 duration=7.189703ms 2024-03-04T15:41:59.038 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=ngalert.multiorg.alertmanager t=2024-03-04T15:41:58.746816976Z level=info msg="starting MultiOrg Alertmanager" 2024-03-04T15:41:59.038 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=ticker t=2024-03-04T15:41:58.746835631Z level=info msg=starting first_tick=2024-03-04T15:42:00Z 2024-03-04T15:41:59.038 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=sqlstore.transactions t=2024-03-04T15:41:58.757425152Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-04T15:41:59.038 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=sqlstore.transactions t=2024-03-04T15:41:58.760296452Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-04T15:41:59.038 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=sqlstore.transactions t=2024-03-04T15:41:58.771440192Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-04T15:41:59.038 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=sqlstore.transactions t=2024-03-04T15:41:58.782930755Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-04T15:41:59.038 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:41:58 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a[38857]: logger=sqlstore.transactions t=2024-03-04T15:41:58.793291962Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-03-04T15:42:00.439 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:00 smithi080 ceph-mon[27564]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:00.439 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:00 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:00.439 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:00 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:42:00.600 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:00 smithi005 ceph-mon[25858]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:00.600 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:00 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:00.600 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:00 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:42:00.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:00 smithi087 ceph-mon[27664]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:00.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:00 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:00.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:00 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:42:01.071 INFO:teuthology.orchestra.run.smithi005.stdout:HOST ADDR LABELS STATUS 2024-03-04T15:42:01.071 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 172.21.15.5 2024-03-04T15:42:01.071 INFO:teuthology.orchestra.run.smithi005.stdout:smithi080 172.21.15.80 2024-03-04T15:42:01.071 INFO:teuthology.orchestra.run.smithi005.stdout:smithi087 172.21.15.87 2024-03-04T15:42:01.071 INFO:teuthology.orchestra.run.smithi005.stdout:3 hosts in cluster 2024-03-04T15:42:01.712 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-04T15:42:01.975 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:42:02.096 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:01 smithi005 bash[51698]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-04T15:42:02.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:02 smithi005 ceph-mon[25858]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:02.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:02 smithi005 ceph-mon[25858]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:42:02.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:02 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:02.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:02 smithi080 ceph-mon[27564]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:02.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:02 smithi080 ceph-mon[27564]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:42:02.597 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:02 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:02.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:02 smithi087 ceph-mon[27664]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:02.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:02 smithi087 ceph-mon[27664]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:42:02.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:02 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:03.681 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:03 smithi005 bash[51698]: Getting image source signatures 2024-03-04T15:42:03.681 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:03 smithi005 bash[51698]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-04T15:42:03.681 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:03 smithi005 bash[51698]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-04T15:42:03.681 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:03 smithi005 bash[51698]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-04T15:42:03.812 INFO:teuthology.orchestra.run.smithi005.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-04T15:42:03.812 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300EW480DGN 447G No 15s ago Has a FileSystem, LVM detected 2024-03-04T15:42:03.812 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 /dev/nvme1n1 ssd Linux_1e119cb573918dece0e3 89.4G Yes 15s ago 2024-03-04T15:42:03.812 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 /dev/nvme2n1 ssd Linux_c9964ea2e74c4abd4603 89.4G Yes 15s ago 2024-03-04T15:42:03.812 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 /dev/nvme3n1 ssd Linux_7f5456628a5167fe05c3 89.4G Yes 15s ago 2024-03-04T15:42:03.812 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 /dev/nvme4n1 ssd Linux_5935cc579deaf4a58ac6 89.4G No 15s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-04T15:42:03.812 INFO:teuthology.orchestra.run.smithi005.stdout:smithi005 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W4HQ1L 931G No 15s ago Has partitions 2024-03-04T15:42:03.812 INFO:teuthology.orchestra.run.smithi005.stdout:smithi080 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204014T400BGN 372G No 0s ago Has a FileSystem, LVM detected 2024-03-04T15:42:03.812 INFO:teuthology.orchestra.run.smithi005.stdout:smithi080 /dev/nvme1n1 ssd Linux_cace3921c89ec99838a6 89.4G Yes 0s ago 2024-03-04T15:42:03.812 INFO:teuthology.orchestra.run.smithi005.stdout:smithi080 /dev/nvme2n1 ssd Linux_8ad86045fb201c52a8cf 89.4G Yes 0s ago 2024-03-04T15:42:03.813 INFO:teuthology.orchestra.run.smithi005.stdout:smithi080 /dev/nvme3n1 ssd Linux_b80fa812465f5ff56a52 89.4G Yes 0s ago 2024-03-04T15:42:03.813 INFO:teuthology.orchestra.run.smithi005.stdout:smithi080 /dev/nvme4n1 ssd Linux_1f4db6893dd4e45e92cc 89.4G No 0s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-04T15:42:03.813 INFO:teuthology.orchestra.run.smithi005.stdout:smithi080 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5R0PW 931G No 0s ago Has partitions 2024-03-04T15:42:03.813 INFO:teuthology.orchestra.run.smithi005.stdout:smithi087 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NQ400BGN 372G No 18s ago Has a FileSystem, LVM detected 2024-03-04T15:42:03.813 INFO:teuthology.orchestra.run.smithi005.stdout:smithi087 /dev/nvme1n1 ssd Linux_d88c439eac4861eb727a 89.4G Yes 18s ago 2024-03-04T15:42:03.813 INFO:teuthology.orchestra.run.smithi005.stdout:smithi087 /dev/nvme2n1 ssd Linux_95379ce2941eeb5812d0 89.4G Yes 18s ago 2024-03-04T15:42:03.813 INFO:teuthology.orchestra.run.smithi005.stdout:smithi087 /dev/nvme3n1 ssd Linux_7213ecafec5bce822b75 89.4G Yes 18s ago 2024-03-04T15:42:03.813 INFO:teuthology.orchestra.run.smithi005.stdout:smithi087 /dev/nvme4n1 ssd Linux_787da765e0afec5d3f29 89.4G No 18s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-04T15:42:03.813 INFO:teuthology.orchestra.run.smithi005.stdout:smithi087 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5LGPP 931G No 18s ago Has partitions 2024-03-04T15:42:04.512 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:04 smithi087 ceph-mon[27664]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:04.512 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:04 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:04.512 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:04 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:04.512 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:04 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:04.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:04 smithi080 ceph-mon[27564]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:04.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:04 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:04.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:04 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:04.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:04 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:04.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:04 smithi005 ceph-mon[25858]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:04.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:04 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:04.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:04 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:04.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:04 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:04.945 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-04T15:42:05.553 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:42:05.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:05 smithi080 ceph-mon[27564]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:42:05.601 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:05 smithi005 ceph-mon[25858]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:42:05.685 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:05 smithi087 ceph-mon[27664]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:42:05.877 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:05 smithi005 bash[51698]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-04T15:42:06.181 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:05 smithi005 bash[51698]: Writing manifest to image destination 2024-03-04T15:42:06.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:06 smithi087 ceph-mon[27664]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:06.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:06 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:06.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:06 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:06.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:06 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:06.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:06 smithi080 ceph-mon[27564]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:06.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:06 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:06.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:06 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:06.596 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:06 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:06.667 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:06 smithi005 podman[51698]: 2024-03-04 15:42:06.376300148 +0000 UTC m=+4.603124154 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-04T15:42:06.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:06 smithi005 ceph-mon[25858]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:06.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:06 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:06.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:06 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:06.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:06 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:06.931 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:06 smithi005 podman[51698]: 2024-03-04 15:42:06.665472207 +0000 UTC m=+4.892296199 container create 909423ded1ebc1e2e84775dcc1efbed4f87b09665e026010524a3228d5f10354 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T15:42:07.338 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:06 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: ::ffff:172.21.15.80 - - [04/Mar/2024:15:42:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T15:42:07.633 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 podman[51698]: 2024-03-04 15:42:07.547305098 +0000 UTC m=+5.774129103 container init 909423ded1ebc1e2e84775dcc1efbed4f87b09665e026010524a3228d5f10354 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T15:42:07.633 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 podman[51698]: 2024-03-04 15:42:07.553626131 +0000 UTC m=+5.780450124 container start 909423ded1ebc1e2e84775dcc1efbed4f87b09665e026010524a3228d5f10354 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T15:42:07.633 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.557Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-04T15:42:07.633 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.557Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-04T15:42:07.633 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-04T15:42:07.633 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-04T15:42:07.633 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-04T15:42:07.633 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-04T15:42:07.633 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-04T15:42:07.633 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=arp 2024-03-04T15:42:07.633 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-04T15:42:07.634 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-04T15:42:07.634 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-04T15:42:07.634 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-04T15:42:07.634 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-04T15:42:07.634 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-04T15:42:07.634 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-04T15:42:07.634 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-04T15:42:07.635 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=edac 2024-03-04T15:42:07.635 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-04T15:42:07.635 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-04T15:42:07.635 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-04T15:42:07.635 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-04T15:42:07.635 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-04T15:42:07.635 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-04T15:42:07.635 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-04T15:42:07.635 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-04T15:42:07.635 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-04T15:42:07.635 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-04T15:42:07.635 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-04T15:42:07.636 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-04T15:42:07.636 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-04T15:42:07.636 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-04T15:42:07.636 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-04T15:42:07.636 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-04T15:42:07.636 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=os 2024-03-04T15:42:07.636 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-04T15:42:07.636 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-04T15:42:07.636 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-04T15:42:07.636 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-04T15:42:07.636 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-04T15:42:07.636 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-04T15:42:07.636 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-04T15:42:07.637 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=stat 2024-03-04T15:42:07.637 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-04T15:42:07.637 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-04T15:42:07.637 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-04T15:42:07.637 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=time 2024-03-04T15:42:07.637 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-04T15:42:07.637 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=uname 2024-03-04T15:42:07.637 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-04T15:42:07.637 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-04T15:42:07.637 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.558Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-04T15:42:07.637 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.559Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-04T15:42:07.637 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a[52143]: ts=2024-03-04T15:42:07.559Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-04T15:42:07.637 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 bash[51698]: 909423ded1ebc1e2e84775dcc1efbed4f87b09665e026010524a3228d5f10354 2024-03-04T15:42:07.931 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:42:07 smithi005 systemd[1]: Started Ceph node-exporter.a for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:42:08.095 INFO:teuthology.orchestra.run.smithi005.stdout:service_type: agent 2024-03-04T15:42:08.095 INFO:teuthology.orchestra.run.smithi005.stdout:service_name: agent 2024-03-04T15:42:08.095 INFO:teuthology.orchestra.run.smithi005.stdout:placement: 2024-03-04T15:42:08.095 INFO:teuthology.orchestra.run.smithi005.stdout: host_pattern: '*' 2024-03-04T15:42:08.095 INFO:teuthology.orchestra.run.smithi005.stdout:status: 2024-03-04T15:42:08.095 INFO:teuthology.orchestra.run.smithi005.stdout: created: '2024-03-04T15:41:14.838837Z' 2024-03-04T15:42:08.096 INFO:teuthology.orchestra.run.smithi005.stdout: last_refresh: '2024-03-04T15:41:48.278315Z' 2024-03-04T15:42:08.096 INFO:teuthology.orchestra.run.smithi005.stdout: running: 3 2024-03-04T15:42:08.096 INFO:teuthology.orchestra.run.smithi005.stdout: size: 3 2024-03-04T15:42:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:events: 2024-03-04T15:42:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:- 2024-03-04T15:41:16.320044Z service:agent [INFO] "service was created" 2024-03-04T15:42:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:--- 2024-03-04T15:42:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:service_type: alertmanager 2024-03-04T15:42:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:service_name: alertmanager 2024-03-04T15:42:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:placement: 2024-03-04T15:42:08.096 INFO:teuthology.orchestra.run.smithi005.stdout: count: 1 2024-03-04T15:42:08.096 INFO:teuthology.orchestra.run.smithi005.stdout: hosts: 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: - smithi005=a 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout:status: 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: created: '2024-03-04T15:40:49.453250Z' 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: last_refresh: '2024-03-04T15:41:48.278550Z' 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: ports: 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: - 9093 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: - 9094 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: running: 1 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: size: 1 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout:events: 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout:- 2024-03-04T15:41:38.657367Z service:alertmanager [INFO] "service was created" 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout:--- 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout:service_type: grafana 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout:service_name: grafana 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout:placement: 2024-03-04T15:42:08.097 INFO:teuthology.orchestra.run.smithi005.stdout: count: 1 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: hosts: 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: - smithi080=a 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout:spec: 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: anonymous_access: true 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: protocol: https 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout:status: 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: created: '2024-03-04T15:40:52.184551Z' 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: last_refresh: '2024-03-04T15:42:03.659278Z' 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: ports: 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: - 3000 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: running: 1 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout: size: 1 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout:events: 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout:- 2024-03-04T15:41:57.342405Z service:grafana [INFO] "service was created" 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout:--- 2024-03-04T15:42:08.098 INFO:teuthology.orchestra.run.smithi005.stdout:service_type: mgr 2024-03-04T15:42:08.099 INFO:teuthology.orchestra.run.smithi005.stdout:service_name: mgr 2024-03-04T15:42:08.099 INFO:teuthology.orchestra.run.smithi005.stdout:placement: 2024-03-04T15:42:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: count: 3 2024-03-04T15:42:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: hosts: 2024-03-04T15:42:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: - smithi005=y 2024-03-04T15:42:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: - smithi080=x 2024-03-04T15:42:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: - smithi087=z 2024-03-04T15:42:08.099 INFO:teuthology.orchestra.run.smithi005.stdout:status: 2024-03-04T15:42:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: created: '2024-03-04T15:39:05.726678Z' 2024-03-04T15:42:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: last_refresh: '2024-03-04T15:41:48.278258Z' 2024-03-04T15:42:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: running: 3 2024-03-04T15:42:08.099 INFO:teuthology.orchestra.run.smithi005.stdout: size: 3 2024-03-04T15:42:08.099 INFO:teuthology.orchestra.run.smithi005.stdout:--- 2024-03-04T15:42:08.099 INFO:teuthology.orchestra.run.smithi005.stdout:service_type: mon 2024-03-04T15:42:08.099 INFO:teuthology.orchestra.run.smithi005.stdout:service_name: mon 2024-03-04T15:42:08.100 INFO:teuthology.orchestra.run.smithi005.stdout:placement: 2024-03-04T15:42:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: count: 3 2024-03-04T15:42:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: hosts: 2024-03-04T15:42:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: - smithi005:172.21.15.5=a 2024-03-04T15:42:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: - smithi080:172.21.15.80=b 2024-03-04T15:42:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: - smithi087:172.21.15.87=c 2024-03-04T15:42:08.100 INFO:teuthology.orchestra.run.smithi005.stdout:status: 2024-03-04T15:42:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: created: '2024-03-04T15:38:38.131827Z' 2024-03-04T15:42:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: last_refresh: '2024-03-04T15:41:48.278161Z' 2024-03-04T15:42:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: running: 3 2024-03-04T15:42:08.100 INFO:teuthology.orchestra.run.smithi005.stdout: size: 3 2024-03-04T15:42:08.100 INFO:teuthology.orchestra.run.smithi005.stdout:--- 2024-03-04T15:42:08.100 INFO:teuthology.orchestra.run.smithi005.stdout:service_type: node-exporter 2024-03-04T15:42:08.100 INFO:teuthology.orchestra.run.smithi005.stdout:service_name: node-exporter 2024-03-04T15:42:08.100 INFO:teuthology.orchestra.run.smithi005.stdout:placement: 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: count: 3 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: hosts: 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: - smithi005=a 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: - smithi080=b 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: - smithi087=c 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout:status: 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: created: '2024-03-04T15:40:46.781438Z' 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: ports: 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: - 9100 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: running: 0 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout: size: 3 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout:events: 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout:- 2024-03-04T15:42:07.731825Z service:node-exporter [INFO] "service was created" 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout:--- 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout:service_type: osd 2024-03-04T15:42:08.101 INFO:teuthology.orchestra.run.smithi005.stdout:service_name: osd 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout:unmanaged: true 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout:spec: 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: filter_logic: AND 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: objectstore: bluestore 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout:status: 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: container_image_id: 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: last_refresh: '2024-03-04T15:41:48.278394Z' 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: running: 3 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: size: 3 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout:--- 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout:service_type: prometheus 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout:service_name: prometheus 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout:placement: 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: count: 1 2024-03-04T15:42:08.102 INFO:teuthology.orchestra.run.smithi005.stdout: hosts: 2024-03-04T15:42:08.103 INFO:teuthology.orchestra.run.smithi005.stdout: - smithi080=a 2024-03-04T15:42:08.103 INFO:teuthology.orchestra.run.smithi005.stdout:status: 2024-03-04T15:42:08.103 INFO:teuthology.orchestra.run.smithi005.stdout: created: '2024-03-04T15:40:44.167565Z' 2024-03-04T15:42:08.103 INFO:teuthology.orchestra.run.smithi005.stdout: last_refresh: '2024-03-04T15:42:03.659198Z' 2024-03-04T15:42:08.103 INFO:teuthology.orchestra.run.smithi005.stdout: ports: 2024-03-04T15:42:08.103 INFO:teuthology.orchestra.run.smithi005.stdout: - 9095 2024-03-04T15:42:08.103 INFO:teuthology.orchestra.run.smithi005.stdout: running: 1 2024-03-04T15:42:08.103 INFO:teuthology.orchestra.run.smithi005.stdout: size: 1 2024-03-04T15:42:08.103 INFO:teuthology.orchestra.run.smithi005.stdout:--- 2024-03-04T15:42:08.103 INFO:teuthology.orchestra.run.smithi005.stdout:service_type: rgw 2024-03-04T15:42:08.103 INFO:teuthology.orchestra.run.smithi005.stdout:service_id: foo.a 2024-03-04T15:42:08.103 INFO:teuthology.orchestra.run.smithi005.stdout:service_name: rgw.foo.a 2024-03-04T15:42:08.104 INFO:teuthology.orchestra.run.smithi005.stdout:placement: 2024-03-04T15:42:08.104 INFO:teuthology.orchestra.run.smithi005.stdout: count: 1 2024-03-04T15:42:08.104 INFO:teuthology.orchestra.run.smithi005.stdout: hosts: 2024-03-04T15:42:08.104 INFO:teuthology.orchestra.run.smithi005.stdout: - smithi005=foo.a 2024-03-04T15:42:08.104 INFO:teuthology.orchestra.run.smithi005.stdout:status: 2024-03-04T15:42:08.104 INFO:teuthology.orchestra.run.smithi005.stdout: created: '2024-03-04T15:40:44.221173Z' 2024-03-04T15:42:08.104 INFO:teuthology.orchestra.run.smithi005.stdout: last_refresh: '2024-03-04T15:41:48.278472Z' 2024-03-04T15:42:08.104 INFO:teuthology.orchestra.run.smithi005.stdout: ports: 2024-03-04T15:42:08.104 INFO:teuthology.orchestra.run.smithi005.stdout: - 80 2024-03-04T15:42:08.104 INFO:teuthology.orchestra.run.smithi005.stdout: running: 1 2024-03-04T15:42:08.104 INFO:teuthology.orchestra.run.smithi005.stdout: size: 1 2024-03-04T15:42:08.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:08 smithi005 ceph-mon[25858]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:08.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:08 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:08.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:08 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:08.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:08 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:08.589 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:08 smithi080 ceph-mon[27564]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:08.589 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:08 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:08.589 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:08 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:08.589 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:08 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:08.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:08 smithi087 ceph-mon[27664]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:08.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:08 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:08.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:08 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:08.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:08 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:08.846 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:08 smithi080 systemd[1]: Starting Ceph node-exporter.b for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:42:09.346 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:08 smithi080 bash[39870]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-04T15:42:09.383 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-04T15:42:09.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:09 smithi087 ceph-mon[27664]: Deploying daemon node-exporter.b on smithi080 2024-03-04T15:42:09.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:09 smithi087 ceph-mon[27664]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-04T15:42:09.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:09 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:09.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:09 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:09.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:09 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:09.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:09 smithi005 ceph-mon[25858]: Deploying daemon node-exporter.b on smithi080 2024-03-04T15:42:09.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:09 smithi005 ceph-mon[25858]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-04T15:42:09.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:09 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:09.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:09 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:09.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:09 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:09.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:09 smithi080 ceph-mon[27564]: Deploying daemon node-exporter.b on smithi080 2024-03-04T15:42:09.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:09 smithi080 ceph-mon[27564]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-04T15:42:09.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:09 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:09.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:09 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:09.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:09 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:09.848 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:09 smithi080 bash[39870]: Getting image source signatures 2024-03-04T15:42:09.848 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:09 smithi080 bash[39870]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-04T15:42:09.848 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:09 smithi080 bash[39870]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-04T15:42:09.848 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:09 smithi080 bash[39870]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-04T15:42:10.279 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:42:10.664 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:10 smithi005 ceph-mon[25858]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:10.748 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:10 smithi080 ceph-mon[27564]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:10.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:10 smithi087 ceph-mon[27664]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:11.070 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:10 smithi080 bash[39870]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-04T15:42:11.346 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 bash[39870]: Writing manifest to image destination 2024-03-04T15:42:11.693 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 podman[39870]: 2024-03-04 15:42:11.316136826 +0000 UTC m=+2.447120999 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-04T15:42:11.693 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 podman[39870]: 2024-03-04 15:42:11.397773713 +0000 UTC m=+2.528757863 container create 1452eb59b4b7b331b7c8a52ad608194820b42b1043f2f00384170a8fc19733db (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T15:42:11.693 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 podman[39870]: 2024-03-04 15:42:11.600018241 +0000 UTC m=+2.731002404 container init 1452eb59b4b7b331b7c8a52ad608194820b42b1043f2f00384170a8fc19733db (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T15:42:11.693 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 podman[39870]: 2024-03-04 15:42:11.605300895 +0000 UTC m=+2.736285053 container start 1452eb59b4b7b331b7c8a52ad608194820b42b1043f2f00384170a8fc19733db (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T15:42:11.693 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.608Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-04T15:42:11.694 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.608Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-04T15:42:11.694 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.609Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-04T15:42:11.694 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.609Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-04T15:42:11.694 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.609Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-04T15:42:11.694 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.609Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-04T15:42:11.694 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-04T15:42:11.694 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=arp 2024-03-04T15:42:11.694 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-04T15:42:11.694 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-04T15:42:11.694 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-04T15:42:11.695 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-04T15:42:11.695 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-04T15:42:11.695 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-04T15:42:11.695 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-04T15:42:11.695 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-04T15:42:11.695 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=edac 2024-03-04T15:42:11.695 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-04T15:42:11.695 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-04T15:42:11.695 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-04T15:42:11.695 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-04T15:42:11.696 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-04T15:42:11.696 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-04T15:42:11.696 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-04T15:42:11.696 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-04T15:42:11.696 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-04T15:42:11.696 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-04T15:42:11.696 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-04T15:42:11.696 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-04T15:42:11.696 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-04T15:42:11.696 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-04T15:42:11.696 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-04T15:42:11.696 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-04T15:42:11.696 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=os 2024-03-04T15:42:11.697 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-04T15:42:11.697 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-04T15:42:11.697 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-04T15:42:11.697 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-04T15:42:11.697 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-04T15:42:11.697 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-04T15:42:11.697 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-04T15:42:11.697 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=stat 2024-03-04T15:42:11.697 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-04T15:42:11.697 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-04T15:42:11.697 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-04T15:42:11.697 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=time 2024-03-04T15:42:11.697 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-04T15:42:11.697 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=uname 2024-03-04T15:42:11.698 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-04T15:42:11.698 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-04T15:42:11.698 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-04T15:42:11.698 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-04T15:42:11.698 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b[39931]: ts=2024-03-04T15:42:11.610Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-04T15:42:11.698 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 bash[39870]: 1452eb59b4b7b331b7c8a52ad608194820b42b1043f2f00384170a8fc19733db 2024-03-04T15:42:12.100 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:42:11 smithi080 systemd[1]: Started Ceph node-exporter.b for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:42:12.490 INFO:teuthology.orchestra.run.smithi005.stdout:osd 3 8s ago - 2024-03-04T15:42:12.652 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:12 smithi087 ceph-mon[27664]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:12.652 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:12 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:12.652 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:12 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:12.652 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:12 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:12.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:12 smithi005 ceph-mon[25858]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:12.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:12 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:12.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:12 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:12.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:12 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:12.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:12 smithi080 ceph-mon[27564]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:12.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:12 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:12.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:12 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:12.847 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:12 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:12.947 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:12 smithi087 systemd[1]: Starting Ceph node-exporter.c for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:42:13.277 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:12 smithi087 bash[40079]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-04T15:42:13.415 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-04T15:42:13.425 INFO:tasks.cephadm:Teardown begin 2024-03-04T15:42:13.425 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:42:13.458 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:42:13.494 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:42:13.528 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-04T15:42:13.528 DEBUG:teuthology.orchestra.run.smithi005:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T15:42:13.543 DEBUG:teuthology.orchestra.run.smithi080:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T15:42:13.562 DEBUG:teuthology.orchestra.run.smithi087:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T15:42:13.583 INFO:tasks.cephadm:Stopping all daemons... 2024-03-04T15:42:13.583 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-04T15:42:13.583 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mon.a 2024-03-04T15:42:13.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:13 smithi005 ceph-mon[25858]: Deploying daemon node-exporter.c on smithi087 2024-03-04T15:42:13.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:13 smithi005 ceph-mon[25858]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:42:13.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:13 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:13.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:13 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:13.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:13 smithi005 ceph-mon[25858]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:13.631 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:13 smithi005 systemd[1]: Stopping Ceph mon.a for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:42:13.777 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:13 smithi087 bash[40079]: Getting image source signatures 2024-03-04T15:42:13.778 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:13 smithi087 bash[40079]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-04T15:42:13.778 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:13 smithi087 bash[40079]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-04T15:42:13.778 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:13 smithi087 bash[40079]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-04T15:42:13.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:13 smithi087 ceph-mon[27664]: Deploying daemon node-exporter.c on smithi087 2024-03-04T15:42:13.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:13 smithi087 ceph-mon[27664]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:42:13.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:13 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:13.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:13 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:13.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:13 smithi087 ceph-mon[27664]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:13.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:13 smithi080 ceph-mon[27564]: Deploying daemon node-exporter.c on smithi087 2024-03-04T15:42:13.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:13 smithi080 ceph-mon[27564]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:42:13.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:13 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:13.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:13 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:13.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:13 smithi080 ceph-mon[27564]: from='mgr.14430 172.21.15.5:0/3091965926' entity='mgr.y' 2024-03-04T15:42:14.574 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:14 smithi005 ceph-mon[25858]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:14.777 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:14 smithi087 ceph-mon[27664]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:14.842 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:14 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-a[25835]: 2024-03-04T15:42:14.571+0000 7f24848cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:42:14.843 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:14 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-a[25835]: 2024-03-04T15:42:14.571+0000 7f24848cd700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-04T15:42:14.843 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:14 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:42:14] ENGINE Bus STOPPING 2024-03-04T15:42:14.843 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:14 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:42:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T15:42:14.843 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:14 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:42:14] ENGINE Bus STOPPED 2024-03-04T15:42:14.843 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:14 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:42:14] ENGINE Bus STARTING 2024-03-04T15:42:14.843 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:14 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:42:14] ENGINE Serving on http://:::9283 2024-03-04T15:42:14.844 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:14 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:42:14] ENGINE Bus STARTED 2024-03-04T15:42:14.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:14 smithi080 ceph-mon[27564]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:42:15.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:14 smithi005 podman[53002]: 2024-03-04 15:42:14.962739817 +0000 UTC m=+0.849882809 container died c077183e60f8a64973c94f9fb41f5cf34bc1c1088f3b71335aed0f8ec3a42d66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T15:42:15.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:15 smithi005 podman[53002]: 2024-03-04 15:42:15.455629646 +0000 UTC m=+1.342772646 container cleanup c077183e60f8a64973c94f9fb41f5cf34bc1c1088f3b71335aed0f8ec3a42d66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-04T15:42:15.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:15 smithi005 bash[53002]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-a 2024-03-04T15:42:15.778 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:15 smithi087 bash[40079]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-04T15:42:15.778 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:15 smithi087 bash[40079]: Writing manifest to image destination 2024-03-04T15:42:15.931 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 04 15:42:15 smithi005 podman[53058]: 2024-03-04 15:42:15.657024456 +0000 UTC m=+1.019234353 container remove c077183e60f8a64973c94f9fb41f5cf34bc1c1088f3b71335aed0f8ec3a42d66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-04T15:42:16.074 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:15 smithi087 podman[40079]: 2024-03-04 15:42:15.747663421 +0000 UTC m=+2.893631492 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-04T15:42:16.074 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:15 smithi087 podman[40079]: 2024-03-04 15:42:15.846215592 +0000 UTC m=+2.992183663 container create fb7fc739f06267972a0e8c7e53c27b2a43b200fbe8bfccc2d5c485fda9184fce (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-04T15:42:16.074 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 podman[40079]: 2024-03-04 15:42:16.073107995 +0000 UTC m=+3.219076081 container init fb7fc739f06267972a0e8c7e53c27b2a43b200fbe8bfccc2d5c485fda9184fce (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-04T15:42:16.528 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 podman[40079]: 2024-03-04 15:42:16.078716282 +0000 UTC m=+3.224684354 container start fb7fc739f06267972a0e8c7e53c27b2a43b200fbe8bfccc2d5c485fda9184fce (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-04T15:42:16.528 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.083Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-04T15:42:16.528 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.083Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-04T15:42:16.528 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.083Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-04T15:42:16.528 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.083Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-04T15:42:16.528 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.083Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-04T15:42:16.528 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.083Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-04T15:42:16.528 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-04T15:42:16.528 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=arp 2024-03-04T15:42:16.529 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-04T15:42:16.529 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-04T15:42:16.529 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-04T15:42:16.529 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-04T15:42:16.529 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-04T15:42:16.529 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-04T15:42:16.529 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-04T15:42:16.529 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-04T15:42:16.529 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=edac 2024-03-04T15:42:16.529 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-04T15:42:16.529 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-04T15:42:16.530 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-04T15:42:16.530 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-04T15:42:16.530 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-04T15:42:16.530 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-04T15:42:16.530 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-04T15:42:16.530 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-04T15:42:16.530 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-04T15:42:16.530 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-04T15:42:16.530 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-04T15:42:16.530 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-04T15:42:16.531 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-04T15:42:16.531 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-04T15:42:16.531 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-04T15:42:16.531 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-04T15:42:16.531 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=os 2024-03-04T15:42:16.531 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-04T15:42:16.531 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-04T15:42:16.531 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-04T15:42:16.531 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-04T15:42:16.531 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-04T15:42:16.532 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-04T15:42:16.532 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-04T15:42:16.532 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=stat 2024-03-04T15:42:16.532 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-04T15:42:16.532 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-04T15:42:16.532 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-04T15:42:16.532 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=time 2024-03-04T15:42:16.532 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-04T15:42:16.533 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=uname 2024-03-04T15:42:16.533 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-04T15:42:16.533 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-04T15:42:16.533 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-04T15:42:16.533 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-04T15:42:16.533 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c[40161]: ts=2024-03-04T15:42:16.084Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-04T15:42:16.533 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 bash[40079]: fb7fc739f06267972a0e8c7e53c27b2a43b200fbe8bfccc2d5c485fda9184fce 2024-03-04T15:42:16.533 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:42:16 smithi087 systemd[1]: Started Ceph node-exporter.c for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:42:16.612 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mon.a.service' 2024-03-04T15:42:16.662 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:42:16.662 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-04T15:42:16.662 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-04T15:42:16.662 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mon.b 2024-03-04T15:42:16.846 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:16 smithi080 systemd[1]: Stopping Ceph mon.b for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:42:17.173 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:17 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-b[27541]: 2024-03-04T15:42:17.129+0000 7fe8953cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:42:17.173 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:17 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-b[27541]: 2024-03-04T15:42:17.129+0000 7fe8953cd700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-04T15:42:17.253 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:16 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: ::ffff:172.21.15.80 - - [04/Mar/2024:15:42:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T15:42:17.254 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:17 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:42:17] ENGINE Bus STOPPING 2024-03-04T15:42:17.254 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:17 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:42:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T15:42:17.254 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:17 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:42:17] ENGINE Bus STOPPED 2024-03-04T15:42:17.254 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:17 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:42:17] ENGINE Bus STARTING 2024-03-04T15:42:17.254 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:17 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:42:17] ENGINE Serving on http://:::9283 2024-03-04T15:42:17.254 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:17 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: [04/Mar/2024:15:42:17] ENGINE Bus STARTED 2024-03-04T15:42:17.464 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:17 smithi080 podman[40224]: 2024-03-04 15:42:17.172093059 +0000 UTC m=+0.144298352 container died 767a142e0f993390ba2731f21152cae5aaf79e754c49ddbd1aa52148217908ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-b, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T15:42:17.805 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:17 smithi080 podman[40224]: 2024-03-04 15:42:17.552950871 +0000 UTC m=+0.525156171 container cleanup 767a142e0f993390ba2731f21152cae5aaf79e754c49ddbd1aa52148217908ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-b, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-04T15:42:17.805 INFO:journalctl@ceph.mon.b.smithi080.stdout:Mar 04 15:42:17 smithi080 bash[40224]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-b 2024-03-04T15:42:17.930 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mon.b.service' 2024-03-04T15:42:17.976 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:42:17.977 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-04T15:42:17.977 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-04T15:42:17.977 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mon.c 2024-03-04T15:42:18.272 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:18 smithi087 systemd[1]: Stopping Ceph mon.c for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:42:18.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:18 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-c[27641]: 2024-03-04T15:42:18.269+0000 7f4ff8d84700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:42:18.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:18 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-c[27641]: 2024-03-04T15:42:18.269+0000 7f4ff8d84700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-04T15:42:18.827 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:18 smithi087 podman[40238]: 2024-03-04 15:42:18.526709245 +0000 UTC m=+0.352152574 container died 22502b7796bd1d7de21140b603c97f5dba36dbf115b0db40a94a13df392113a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-04T15:42:19.161 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:18 smithi087 podman[40238]: 2024-03-04 15:42:18.925467465 +0000 UTC m=+0.750910788 container cleanup 22502b7796bd1d7de21140b603c97f5dba36dbf115b0db40a94a13df392113a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-c, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T15:42:19.161 INFO:journalctl@ceph.mon.c.smithi087.stdout:Mar 04 15:42:18 smithi087 bash[40238]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mon-c 2024-03-04T15:42:19.300 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mon.c.service' 2024-03-04T15:42:19.344 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:42:19.344 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-04T15:42:19.344 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-04T15:42:19.344 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mgr.y 2024-03-04T15:42:19.643 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:19 smithi005 systemd[1]: Stopping Ceph mgr.y for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:42:19.931 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:19 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:42:19.641+0000 7fce97abd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:42:19.931 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:19 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y[26109]: 2024-03-04T15:42:19.641+0000 7fce97abd700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-04T15:42:19.931 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:19 smithi005 podman[53283]: 2024-03-04 15:42:19.672635017 +0000 UTC m=+0.138887291 container died f139651d923a0a0e3d3e1b83b3e1f4540adfe0c7b76de056a8c8d10d6dc84da5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222) 2024-03-04T15:42:20.383 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:20 smithi005 podman[53283]: 2024-03-04 15:42:20.138654035 +0000 UTC m=+0.604906308 container cleanup f139651d923a0a0e3d3e1b83b3e1f4540adfe0c7b76de056a8c8d10d6dc84da5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222) 2024-03-04T15:42:20.383 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 04 15:42:20 smithi005 bash[53283]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-y 2024-03-04T15:42:20.505 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mgr.y.service' 2024-03-04T15:42:20.541 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:42:20.541 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-04T15:42:20.542 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-04T15:42:20.542 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mgr.x 2024-03-04T15:42:20.846 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:42:20 smithi080 systemd[1]: Stopping Ceph mgr.x for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:42:22.347 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:42:22 smithi080 podman[40355]: 2024-03-04 15:42:22.231061725 +0000 UTC m=+1.496979015 container died b03f3786212171e32d9c4acc8b0048f396b6fe0a274654d0edc9ee570cd528f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T15:42:26.280 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:42:25 smithi080 podman[40355]: 2024-03-04 15:42:25.994613945 +0000 UTC m=+5.260531235 container cleanup b03f3786212171e32d9c4acc8b0048f396b6fe0a274654d0edc9ee570cd528f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T15:42:26.280 INFO:journalctl@ceph.mgr.x.smithi080.stdout:Mar 04 15:42:25 smithi080 bash[40355]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-x 2024-03-04T15:42:26.434 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mgr.x.service' 2024-03-04T15:42:26.474 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:42:26.475 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-04T15:42:26.475 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-04T15:42:26.475 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mgr.z 2024-03-04T15:42:26.778 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:42:26 smithi087 systemd[1]: Stopping Ceph mgr.z for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:42:27.154 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:42:26 smithi087 podman[40818]: 2024-03-04 15:42:26.834394435 +0000 UTC m=+0.157943855 container died 4f153c08506c60edbcb026234cffa6034ad5b9c6508aaff71de0e6c0a578b0de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-03-04T15:42:27.527 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:42:27 smithi087 podman[40818]: 2024-03-04 15:42:27.243929772 +0000 UTC m=+0.567479199 container cleanup 4f153c08506c60edbcb026234cffa6034ad5b9c6508aaff71de0e6c0a578b0de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T15:42:27.527 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:42:27 smithi087 bash[40818]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z 2024-03-04T15:42:28.277 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:42:28 smithi087 podman[40857]: 2024-03-04 15:42:28.056939573 +0000 UTC m=+1.215201901 container remove 4f153c08506c60edbcb026234cffa6034ad5b9c6508aaff71de0e6c0a578b0de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-mgr-z, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5) 2024-03-04T15:42:28.277 INFO:journalctl@ceph.mgr.z.smithi087.stdout:Mar 04 15:42:28 smithi087 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-04T15:42:28.314 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@mgr.z.service' 2024-03-04T15:42:28.352 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:42:28.352 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-04T15:42:28.352 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-04T15:42:28.352 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-97c626fc-da3b-11ee-95c6-87774f69a715@osd.0 2024-03-04T15:42:28.654 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:42:28 smithi005 systemd[1]: Stopping Ceph osd.0 for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:42:29.650 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:42:29 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-0[38313]: 2024-03-04T15:42:29.443+0000 7f291301c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:42:29.650 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:42:29 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-0[38313]: 2024-03-04T15:42:29.443+0000 7f291301c700 -1 osd.0 32 *** Got signal Terminated *** 2024-03-04T15:42:29.651 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:42:29 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-0[38313]: 2024-03-04T15:42:29.443+0000 7f291301c700 -1 osd.0 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T15:42:35.070 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:42:34 smithi005 podman[53837]: 2024-03-04 15:42:34.802662013 +0000 UTC m=+5.492055234 container died de20379aebe787f505f5c06527a81f1c8c1589b9b91ab556abd1a2d03bb939da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T15:42:35.647 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:42:35 smithi005 podman[53837]: 2024-03-04 15:42:35.386181978 +0000 UTC m=+6.075575199 container cleanup de20379aebe787f505f5c06527a81f1c8c1589b9b91ab556abd1a2d03bb939da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-03-04T15:42:35.647 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:42:35 smithi005 bash[53837]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-0 2024-03-04T15:42:35.931 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:42:35 smithi005 podman[53984]: 2024-03-04 15:42:35.645937005 +0000 UTC m=+0.843438281 container remove de20379aebe787f505f5c06527a81f1c8c1589b9b91ab556abd1a2d03bb939da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-04T15:42:36.805 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:42:36 smithi005 podman[54175]: 2024-03-04 15:42:36.510192562 +0000 UTC m=+0.112693004 container create c9b17bdb7352c9a84550c94ac712e11c04b6401908c4470da325bcd8b7755ae0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=) 2024-03-04T15:42:36.805 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:42:36 smithi005 podman[54175]: 2024-03-04 15:42:36.415002406 +0000 UTC m=+0.017502820 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:42:36.806 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:42:36 smithi005 podman[54175]: 2024-03-04 15:42:36.72267866 +0000 UTC m=+0.325179063 container init c9b17bdb7352c9a84550c94ac712e11c04b6401908c4470da325bcd8b7755ae0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-04T15:42:36.806 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:42:36 smithi005 podman[54175]: 2024-03-04 15:42:36.727978256 +0000 UTC m=+0.330478657 container start c9b17bdb7352c9a84550c94ac712e11c04b6401908c4470da325bcd8b7755ae0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-04T15:42:36.806 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:42:36 smithi005 podman[54175]: 2024-03-04 15:42:36.803986846 +0000 UTC m=+0.406487257 container attach c9b17bdb7352c9a84550c94ac712e11c04b6401908c4470da325bcd8b7755ae0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-04T15:42:37.181 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:42:37 smithi005 podman[54175]: 2024-03-04 15:42:37.026966107 +0000 UTC m=+0.629466499 container died c9b17bdb7352c9a84550c94ac712e11c04b6401908c4470da325bcd8b7755ae0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-0-deactivate, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-04T15:42:37.553 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 04 15:42:37 smithi005 podman[54221]: 2024-03-04 15:42:37.532073745 +0000 UTC m=+0.500454976 container remove c9b17bdb7352c9a84550c94ac712e11c04b6401908c4470da325bcd8b7755ae0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux ) 2024-03-04T15:42:37.559 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@osd.0.service' 2024-03-04T15:42:37.601 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:42:37.601 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-04T15:42:37.601 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-04T15:42:37.601 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-97c626fc-da3b-11ee-95c6-87774f69a715@osd.1 2024-03-04T15:42:37.894 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:42:37 smithi080 systemd[1]: Stopping Ceph osd.1 for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:42:38.346 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:42:37 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-1[33185]: 2024-03-04T15:42:37.891+0000 7ff44c748700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:42:38.346 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:42:37 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-1[33185]: 2024-03-04T15:42:37.891+0000 7ff44c748700 -1 osd.1 32 *** Got signal Terminated *** 2024-03-04T15:42:38.346 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:42:37 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-1[33185]: 2024-03-04T15:42:37.891+0000 7ff44c748700 -1 osd.1 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T15:42:43.692 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:42:43 smithi080 podman[41228]: 2024-03-04 15:42:43.421285389 +0000 UTC m=+5.625799554 container died 6998c7ee4d77c4ed3e8ef78ecc8bb8a149af40ed6f09ae58ab7c3e9702d0b2db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-1, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-04T15:42:44.041 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:42:43 smithi080 podman[41228]: 2024-03-04 15:42:43.781218573 +0000 UTC m=+5.985732725 container cleanup 6998c7ee4d77c4ed3e8ef78ecc8bb8a149af40ed6f09ae58ab7c3e9702d0b2db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-1, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T15:42:44.042 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:42:43 smithi080 bash[41228]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-1 2024-03-04T15:42:44.346 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:42:44 smithi080 podman[41252]: 2024-03-04 15:42:44.04041485 +0000 UTC m=+0.613176108 container remove 6998c7ee4d77c4ed3e8ef78ecc8bb8a149af40ed6f09ae58ab7c3e9702d0b2db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-1, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-04T15:42:44.653 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:42:44 smithi080 podman[41351]: 2024-03-04 15:42:44.257196342 +0000 UTC m=+0.018495186 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:42:44.653 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:42:44 smithi080 podman[41351]: 2024-03-04 15:42:44.36628054 +0000 UTC m=+0.127579372 container create c1984814aa0273ab1a3df2cea83c155308eeef5ea2ac5fdb7f0d4428cda686e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-04T15:42:44.654 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:42:44 smithi080 podman[41351]: 2024-03-04 15:42:44.560271532 +0000 UTC m=+0.321570363 container init c1984814aa0273ab1a3df2cea83c155308eeef5ea2ac5fdb7f0d4428cda686e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-04T15:42:44.654 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:42:44 smithi080 podman[41351]: 2024-03-04 15:42:44.565994491 +0000 UTC m=+0.327293332 container start c1984814aa0273ab1a3df2cea83c155308eeef5ea2ac5fdb7f0d4428cda686e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-04T15:42:44.654 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:42:44 smithi080 podman[41351]: 2024-03-04 15:42:44.651922255 +0000 UTC m=+0.413221098 container attach c1984814aa0273ab1a3df2cea83c155308eeef5ea2ac5fdb7f0d4428cda686e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T15:42:45.096 INFO:journalctl@ceph.osd.1.smithi080.stdout:Mar 04 15:42:44 smithi080 podman[41351]: 2024-03-04 15:42:44.856588662 +0000 UTC m=+0.617887493 container died c1984814aa0273ab1a3df2cea83c155308eeef5ea2ac5fdb7f0d4428cda686e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222) 2024-03-04T15:42:45.447 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@osd.1.service' 2024-03-04T15:42:45.490 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:42:45.490 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-04T15:42:45.491 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-04T15:42:45.491 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-97c626fc-da3b-11ee-95c6-87774f69a715@osd.2 2024-03-04T15:42:45.777 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:42:45 smithi087 systemd[1]: Stopping Ceph osd.2 for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:42:47.436 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:42:47 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-2[34276]: 2024-03-04T15:42:47.027+0000 7f2a4100c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:42:47.437 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:42:47 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-2[34276]: 2024-03-04T15:42:47.027+0000 7f2a4100c700 -1 osd.2 32 *** Got signal Terminated *** 2024-03-04T15:42:47.437 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:42:47 smithi087 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-2[34276]: 2024-03-04T15:42:47.027+0000 7f2a4100c700 -1 osd.2 32 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T15:42:52.664 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:42:52 smithi087 podman[41267]: 2024-03-04 15:42:52.386844988 +0000 UTC m=+6.696858849 container died 03f1dd7f9a878196a36d2e958bda5381c56a4e90dab0289ac3800d06a410aac5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux ) 2024-03-04T15:42:53.021 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:42:52 smithi087 podman[41267]: 2024-03-04 15:42:52.744385891 +0000 UTC m=+7.054399744 container cleanup 03f1dd7f9a878196a36d2e958bda5381c56a4e90dab0289ac3800d06a410aac5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2) 2024-03-04T15:42:53.021 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:42:52 smithi087 bash[41267]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-2 2024-03-04T15:42:53.277 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:42:53 smithi087 podman[41604]: 2024-03-04 15:42:53.020345471 +0000 UTC m=+0.637160631 container remove 03f1dd7f9a878196a36d2e958bda5381c56a4e90dab0289ac3800d06a410aac5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 2024-03-04T15:42:53.658 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:42:53 smithi087 podman[41700]: 2024-03-04 15:42:53.264896497 +0000 UTC m=+0.017450513 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:42:53.658 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:42:53 smithi087 podman[41700]: 2024-03-04 15:42:53.387769049 +0000 UTC m=+0.140323100 container create 7515dfac9cf809007524aedbaf2ddd8b77a988ee2e277b31adba5314daf24687 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-03-04T15:42:53.658 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:42:53 smithi087 podman[41700]: 2024-03-04 15:42:53.582383188 +0000 UTC m=+0.334937199 container init 7515dfac9cf809007524aedbaf2ddd8b77a988ee2e277b31adba5314daf24687 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-04T15:42:53.658 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:42:53 smithi087 podman[41700]: 2024-03-04 15:42:53.587922914 +0000 UTC m=+0.340476917 container start 7515dfac9cf809007524aedbaf2ddd8b77a988ee2e277b31adba5314daf24687 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222) 2024-03-04T15:42:53.658 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:42:53 smithi087 podman[41700]: 2024-03-04 15:42:53.656760175 +0000 UTC m=+0.409314178 container attach 7515dfac9cf809007524aedbaf2ddd8b77a988ee2e277b31adba5314daf24687 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-04T15:42:54.027 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:42:53 smithi087 podman[41700]: 2024-03-04 15:42:53.874385157 +0000 UTC m=+0.626939161 container died 7515dfac9cf809007524aedbaf2ddd8b77a988ee2e277b31adba5314daf24687 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-04T15:42:54.403 INFO:journalctl@ceph.osd.2.smithi087.stdout:Mar 04 15:42:54 smithi087 podman[41745]: 2024-03-04 15:42:54.392899247 +0000 UTC m=+0.512820090 container remove 7515dfac9cf809007524aedbaf2ddd8b77a988ee2e277b31adba5314daf24687 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux ) 2024-03-04T15:42:54.409 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@osd.2.service' 2024-03-04T15:42:54.450 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:42:54.451 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-04T15:42:54.451 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-04T15:42:54.451 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-97c626fc-da3b-11ee-95c6-87774f69a715@rgw.foo.a 2024-03-04T15:42:54.752 INFO:journalctl@ceph.rgw.foo.a.smithi005.stdout:Mar 04 15:42:54 smithi005 systemd[1]: Stopping Ceph rgw.foo.a for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:42:55.416 INFO:journalctl@ceph.rgw.foo.a.smithi005.stdout:Mar 04 15:42:55 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-rgw-foo-a[41514]: 2024-03-04T15:42:55.061+0000 7f4c92d1a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:42:55.416 INFO:journalctl@ceph.rgw.foo.a.smithi005.stdout:Mar 04 15:42:55 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-rgw-foo-a[41514]: 2024-03-04T15:42:55.061+0000 7f4ca1592ac0 -1 shutting down 2024-03-04T15:43:05.368 INFO:journalctl@ceph.rgw.foo.a.smithi005.stdout:Mar 04 15:43:05 smithi005 bash[54911]: time="2024-03-04T15:43:05Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-97c626fc-da3b-11ee-95c6-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-04T15:43:05.368 INFO:journalctl@ceph.rgw.foo.a.smithi005.stdout:Mar 04 15:43:05 smithi005 podman[54911]: 2024-03-04 15:43:05.086225922 +0000 UTC m=+10.120521909 container died 3c0a150b7041738a3439e29efb50e2f0e17f304302f17b624f95206ac0116c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-rgw-foo-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T15:43:05.681 INFO:journalctl@ceph.rgw.foo.a.smithi005.stdout:Mar 04 15:43:05 smithi005 podman[54911]: 2024-03-04 15:43:05.582135914 +0000 UTC m=+10.616431891 container cleanup 3c0a150b7041738a3439e29efb50e2f0e17f304302f17b624f95206ac0116c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-04T15:43:05.681 INFO:journalctl@ceph.rgw.foo.a.smithi005.stdout:Mar 04 15:43:05 smithi005 bash[54911]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-rgw-foo-a 2024-03-04T15:43:06.563 INFO:journalctl@ceph.rgw.foo.a.smithi005.stdout:Mar 04 15:43:06 smithi005 bash[55483]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-rgw-foo-a 2024-03-04T15:43:06.568 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@rgw.foo.a.service' 2024-03-04T15:43:06.597 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:43:06.317Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.5:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.5:8765: connect: connection refused" 2024-03-04T15:43:06.597 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:43:06.318Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.5:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.5:8765: connect: connection refused" 2024-03-04T15:43:06.597 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:43:06.318Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.5:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.5:8765: connect: connection refused" 2024-03-04T15:43:06.609 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:43:06.609 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-04T15:43:06.609 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-04T15:43:06.610 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ceph-97c626fc-da3b-11ee-95c6-87774f69a715@prometheus.a 2024-03-04T15:43:06.906 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 systemd[1]: Stopping Ceph prometheus.a for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:43:07.297 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:43:06.904Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-04T15:43:07.298 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:43:06.904Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-04T15:43:07.298 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:43:06.904Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-04T15:43:07.298 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:43:06.904Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-04T15:43:07.298 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:43:06.904Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-04T15:43:07.298 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:43:06.904Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-04T15:43:07.298 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:43:06.904Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-04T15:43:07.298 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:43:06.904Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-04T15:43:07.298 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:43:06.905Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-04T15:43:07.298 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:43:06.905Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-04T15:43:07.298 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:43:06.905Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-04T15:43:07.298 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a[35996]: ts=2024-03-04T15:43:06.905Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-04T15:43:07.298 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:06 smithi080 podman[42253]: 2024-03-04 15:43:06.915473944 +0000 UTC m=+0.112993096 container died 713fb4e36d740818b3492c79e68b212f9105358495ce0a23c3ae54f188921b73 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T15:43:07.596 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:07 smithi080 podman[42253]: 2024-03-04 15:43:07.393706195 +0000 UTC m=+0.591225349 container cleanup 713fb4e36d740818b3492c79e68b212f9105358495ce0a23c3ae54f188921b73 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T15:43:07.597 INFO:journalctl@ceph.prometheus.a.smithi080.stdout:Mar 04 15:43:07 smithi080 bash[42253]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-prometheus-a 2024-03-04T15:43:07.810 DEBUG:teuthology.orchestra.run.smithi080:> sudo pkill -f 'journalctl -f -n 0 -u ceph-97c626fc-da3b-11ee-95c6-87774f69a715@prometheus.a.service' 2024-03-04T15:43:07.848 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:43:07.848 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-04T15:43:07.848 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T15:43:08.044 INFO:teuthology.orchestra.run.smithi005.stdout:Deleting cluster with fsid: 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:43:09.181 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:43:09 smithi005 systemd[1]: Stopping Ceph alertmanager.a for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:43:09.603 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:43:09 smithi005 ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a[49242]: ts=2024-03-04T15:43:09.334Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-04T15:43:09.603 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:43:09 smithi005 podman[55836]: 2024-03-04 15:43:09.355140174 +0000 UTC m=+0.135599739 container died f2f4e735e3ca8b7364dc6d9215b79b42ec15a4943e7c14003eb53c3b2a7d2761 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T15:43:09.931 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:43:09 smithi005 podman[55836]: 2024-03-04 15:43:09.781775786 +0000 UTC m=+0.562235349 container cleanup f2f4e735e3ca8b7364dc6d9215b79b42ec15a4943e7c14003eb53c3b2a7d2761 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T15:43:09.931 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:43:09 smithi005 bash[55836]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a 2024-03-04T15:43:10.239 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:43:09 smithi005 podman[55855]: 2024-03-04 15:43:09.966407997 +0000 UTC m=+0.614243267 container remove f2f4e735e3ca8b7364dc6d9215b79b42ec15a4943e7c14003eb53c3b2a7d2761 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T15:43:10.239 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:43:10 smithi005 podman[55855]: 2024-03-04 15:43:10.042015931 +0000 UTC m=+0.689851199 volume remove 7cec5c8741ef60a642fd95c0517270ecdd1ddf1a0216bf11ff682f622e9e5f64 2024-03-04T15:43:10.681 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:43:10 smithi005 systemd[1]: Stopping Ceph node-exporter.a for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:43:10.681 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:43:10 smithi005 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-04T15:43:10.681 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:43:10 smithi005 systemd[1]: Stopped Ceph alertmanager.a for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:43:10.681 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 04 15:43:10 smithi005 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@alertmanager.a.service: Consumed 1.125s CPU time. 2024-03-04T15:43:11.037 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:43:10 smithi005 podman[55970]: 2024-03-04 15:43:10.683332121 +0000 UTC m=+0.101788338 container died 909423ded1ebc1e2e84775dcc1efbed4f87b09665e026010524a3228d5f10354 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T15:43:11.038 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:43:11 smithi005 podman[55970]: 2024-03-04 15:43:11.036036628 +0000 UTC m=+0.454492846 container cleanup 909423ded1ebc1e2e84775dcc1efbed4f87b09665e026010524a3228d5f10354 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T15:43:11.389 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:43:11 smithi005 bash[55970]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a 2024-03-04T15:43:11.664 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:43:11 smithi005 podman[55989]: 2024-03-04 15:43:11.387404283 +0000 UTC m=+0.705532711 container remove 909423ded1ebc1e2e84775dcc1efbed4f87b09665e026010524a3228d5f10354 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T15:43:11.664 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:43:11 smithi005 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-04T15:43:11.664 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:43:11 smithi005 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-04T15:43:11.664 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:43:11 smithi005 systemd[1]: Stopped Ceph node-exporter.a for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:43:11.664 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 04 15:43:11 smithi005 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@node-exporter.a.service: Consumed 1.869s CPU time. 2024-03-04T15:43:12.159 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T15:43:12.340 INFO:teuthology.orchestra.run.smithi080.stdout:Deleting cluster with fsid: 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:43:13.658 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:43:13 smithi080 systemd[1]: Stopping Ceph grafana.a for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:43:13.938 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:43:13 smithi080 podman[42816]: 2024-03-04 15:43:13.65657897 +0000 UTC m=+0.111752208 container died 9508aed1abeea54fbdcc1ae0478c554891062361a013c03d54493be27328330c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, vcs-type=git, architecture=x86_64, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, maintainer=Paul Cuzner , vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=) 2024-03-04T15:43:14.279 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:43:14 smithi080 podman[42816]: 2024-03-04 15:43:14.018445196 +0000 UTC m=+0.473618434 container cleanup 9508aed1abeea54fbdcc1ae0478c554891062361a013c03d54493be27328330c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, version=8.5, com.redhat.component=ubi8-container, vcs-type=git, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, vendor=Red Hat, Inc., architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-03-04T15:43:14.279 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:43:14 smithi080 bash[42816]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a 2024-03-04T15:43:14.596 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:43:14 smithi080 podman[42835]: 2024-03-04 15:43:14.277659574 +0000 UTC m=+0.624171881 container remove 9508aed1abeea54fbdcc1ae0478c554891062361a013c03d54493be27328330c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-grafana-a, io.openshift.tags=base rhel8, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.32.0, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, name=ubi8, description=Ceph Grafana Container, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git) 2024-03-04T15:43:14.596 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:43:14 smithi080 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-04T15:43:14.596 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:43:14 smithi080 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-04T15:43:14.597 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:43:14 smithi080 systemd[1]: Stopped Ceph grafana.a for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:43:14.597 INFO:journalctl@ceph.grafana.a.smithi080.stdout:Mar 04 15:43:14 smithi080 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@grafana.a.service: Consumed 3.527s CPU time. 2024-03-04T15:43:14.921 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:43:14 smithi080 systemd[1]: Stopping Ceph node-exporter.b for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:43:15.191 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:43:14 smithi080 podman[42958]: 2024-03-04 15:43:14.919258491 +0000 UTC m=+0.122284807 container died 1452eb59b4b7b331b7c8a52ad608194820b42b1043f2f00384170a8fc19733db (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T15:43:15.516 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:43:15 smithi080 podman[42958]: 2024-03-04 15:43:15.288759921 +0000 UTC m=+0.491786237 container cleanup 1452eb59b4b7b331b7c8a52ad608194820b42b1043f2f00384170a8fc19733db (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T15:43:15.516 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:43:15 smithi080 bash[42958]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b 2024-03-04T15:43:15.809 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:43:15 smithi080 podman[42982]: 2024-03-04 15:43:15.514801196 +0000 UTC m=+0.597023976 container remove 1452eb59b4b7b331b7c8a52ad608194820b42b1043f2f00384170a8fc19733db (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T15:43:15.809 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:43:15 smithi080 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-04T15:43:15.809 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:43:15 smithi080 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-04T15:43:15.809 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:43:15 smithi080 systemd[1]: Stopped Ceph node-exporter.b for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:43:15.810 INFO:journalctl@ceph.node-exporter.b.smithi080.stdout:Mar 04 15:43:15 smithi080 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@node-exporter.b.service: Consumed 1.778s CPU time. 2024-03-04T15:43:16.319 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T15:43:16.513 INFO:teuthology.orchestra.run.smithi087.stdout:Deleting cluster with fsid: 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:43:17.527 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:43:17 smithi087 systemd[1]: Stopping Ceph node-exporter.c for 97c626fc-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:43:17.960 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:43:17 smithi087 podman[42563]: 2024-03-04 15:43:17.687959449 +0000 UTC m=+0.102311352 container died fb7fc739f06267972a0e8c7e53c27b2a43b200fbe8bfccc2d5c485fda9184fce (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-04T15:43:18.229 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:43:18 smithi087 podman[42563]: 2024-03-04 15:43:18.05763842 +0000 UTC m=+0.471990321 container cleanup fb7fc739f06267972a0e8c7e53c27b2a43b200fbe8bfccc2d5c485fda9184fce (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-04T15:43:18.229 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:43:18 smithi087 bash[42563]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c 2024-03-04T15:43:18.527 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:43:18 smithi087 podman[42580]: 2024-03-04 15:43:18.333945495 +0000 UTC m=+0.650966191 container remove fb7fc739f06267972a0e8c7e53c27b2a43b200fbe8bfccc2d5c485fda9184fce (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-97c626fc-da3b-11ee-95c6-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-04T15:43:18.527 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:43:18 smithi087 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@node-exporter.c.service: Main process exited, code=exited, status=143/n/a 2024-03-04T15:43:18.527 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:43:18 smithi087 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@node-exporter.c.service: Failed with result 'exit-code'. 2024-03-04T15:43:18.527 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:43:18 smithi087 systemd[1]: Stopped Ceph node-exporter.c for 97c626fc-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:43:18.528 INFO:journalctl@ceph.node-exporter.c.smithi087.stdout:Mar 04 15:43:18 smithi087 systemd[1]: ceph-97c626fc-da3b-11ee-95c6-87774f69a715@node-exporter.c.service: Consumed 1.831s CPU time. 2024-03-04T15:43:19.145 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:43:19.180 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:43:19.215 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:43:19.244 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-04T15:43:19.246 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216/remote/smithi005/crash 2024-03-04T15:43:19.247 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/crash -- . 2024-03-04T15:43:19.292 INFO:teuthology.orchestra.run.smithi005.stderr:tar: /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T15:43:19.292 INFO:teuthology.orchestra.run.smithi005.stderr:tar: Error is not recoverable: exiting now 2024-03-04T15:43:19.294 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216/remote/smithi080/crash 2024-03-04T15:43:19.295 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/crash -- . 2024-03-04T15:43:19.333 INFO:teuthology.orchestra.run.smithi080.stderr:tar: /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T15:43:19.333 INFO:teuthology.orchestra.run.smithi080.stderr:tar: Error is not recoverable: exiting now 2024-03-04T15:43:19.335 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216/remote/smithi087/crash 2024-03-04T15:43:19.335 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/crash -- . 2024-03-04T15:43:19.374 INFO:teuthology.orchestra.run.smithi087.stderr:tar: /var/lib/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T15:43:19.374 INFO:teuthology.orchestra.run.smithi087.stderr:tar: Error is not recoverable: exiting now 2024-03-04T15:43:19.375 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-04T15:43:19.376 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-04T15:43:19.407 INFO:teuthology.orchestra.run.smithi005.stdout:2024-03-04T15:38:57.904963+0000 mon.a (mon.0) 248 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T15:43:19.407 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-04T15:43:19.408 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[SEC\]' /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-04T15:43:19.478 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[ERR\]' /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-04T15:43:19.547 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[WRN\]' /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-04T15:43:19.616 INFO:teuthology.orchestra.run.smithi005.stdout:2024-03-04T15:38:57.904963+0000 mon.a (mon.0) 248 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T15:43:19.616 INFO:tasks.cephadm:Compressing logs... 2024-03-04T15:43:19.616 DEBUG:teuthology.orchestra.run.smithi005:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T15:43:19.658 DEBUG:teuthology.orchestra.run.smithi080:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T15:43:19.661 DEBUG:teuthology.orchestra.run.smithi087:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T15:43:19.684 INFO:teuthology.orchestra.run.smithi005.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T15:43:19.684 INFO:teuthology.orchestra.run.smithi005.stderr:: No such file or directory 2024-03-04T15:43:19.684 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mon.a.log 2024-03-04T15:43:19.684 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T15:43:19.684 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-04T15:43:19.684 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mon.a.log: /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mgr.y.log 2024-03-04T15:43:19.684 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-osd.0.log 2024-03-04T15:43:19.686 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-client.rgw.foo.a.log: /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T15:43:19.686 INFO:teuthology.orchestra.run.smithi005.stderr: 57.8% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-04T15:43:19.686 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.log 2024-03-04T15:43:19.687 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T15:43:19.687 INFO:teuthology.orchestra.run.smithi005.stderr: 83.0% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T15:43:19.687 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.log: 89.9% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T15:43:19.687 INFO:teuthology.orchestra.run.smithi005.stderr: 84.9% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T15:43:19.687 INFO:teuthology.orchestra.run.smithi005.stderr: 83.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T15:43:19.689 INFO:teuthology.orchestra.run.smithi087.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T15:43:19.689 INFO:teuthology.orchestra.run.smithi087.stderr:: No such file or directory 2024-03-04T15:43:19.689 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T15:43:19.689 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mgr.z.log 2024-03-04T15:43:19.689 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T15:43:19.689 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.audit.log: /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.log 2024-03-04T15:43:19.692 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-osd.2.log 2024-03-04T15:43:19.692 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log: /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.log: gzip 81.6% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T15:43:19.692 INFO:teuthology.orchestra.run.smithi087.stderr: -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T15:43:19.692 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mon.c.log 2024-03-04T15:43:19.692 INFO:teuthology.orchestra.run.smithi087.stderr: 90.0% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T15:43:19.692 INFO:teuthology.orchestra.run.smithi087.stderr: 84.6% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T15:43:19.692 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-volume.log: 79.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T15:43:19.693 INFO:teuthology.orchestra.run.smithi080.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T15:43:19.694 INFO:teuthology.orchestra.run.smithi080.stderr:: No such file or directory 2024-03-04T15:43:19.694 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T15:43:19.694 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-osd.1.log 2024-03-04T15:43:19.694 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mgr.x.log 2024-03-04T15:43:19.694 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T15:43:19.694 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.log 2024-03-04T15:43:19.694 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log: 90.0% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T15:43:19.694 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T15:43:19.694 INFO:teuthology.orchestra.run.smithi080.stderr: 81.5% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T15:43:19.694 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mon.b.log 2024-03-04T15:43:19.694 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.log: /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-volume.log: /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mon.b.log: 84.4% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T15:43:19.695 INFO:teuthology.orchestra.run.smithi080.stderr: 77.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T15:43:19.695 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mon.c.log: 90.3% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mgr.z.log.gz 2024-03-04T15:43:19.696 INFO:teuthology.orchestra.run.smithi080.stderr: 90.3% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mgr.x.log.gz 2024-03-04T15:43:19.704 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-volume.log: 90.2% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T15:43:19.706 INFO:teuthology.orchestra.run.smithi087.stderr: 90.2% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T15:43:19.709 INFO:teuthology.orchestra.run.smithi080.stderr: 90.2% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T15:43:19.757 INFO:teuthology.orchestra.run.smithi087.stderr: 92.4% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mon.c.log.gz 2024-03-04T15:43:19.758 INFO:teuthology.orchestra.run.smithi080.stderr: 92.4% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mon.b.log.gz 2024-03-04T15:43:19.771 INFO:teuthology.orchestra.run.smithi005.stderr: 88.2% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mgr.y.log.gz 2024-03-04T15:43:19.918 INFO:teuthology.orchestra.run.smithi005.stderr: 91.5% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-mon.a.log.gz 2024-03-04T15:43:19.963 INFO:teuthology.orchestra.run.smithi087.stderr: 94.2% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-osd.2.log.gz 2024-03-04T15:43:19.965 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-03-04T15:43:19.966 INFO:teuthology.orchestra.run.smithi087.stderr:real 0m0.289s 2024-03-04T15:43:19.966 INFO:teuthology.orchestra.run.smithi087.stderr:user 0m0.365s 2024-03-04T15:43:19.966 INFO:teuthology.orchestra.run.smithi087.stderr:sys 0m0.031s 2024-03-04T15:43:19.981 INFO:teuthology.orchestra.run.smithi005.stderr: 94.2% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-osd.0.log.gz 2024-03-04T15:43:19.983 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-04T15:43:19.983 INFO:teuthology.orchestra.run.smithi005.stderr:real 0m0.313s 2024-03-04T15:43:19.983 INFO:teuthology.orchestra.run.smithi005.stderr:user 0m0.632s 2024-03-04T15:43:19.983 INFO:teuthology.orchestra.run.smithi005.stderr:sys 0m0.042s 2024-03-04T15:43:20.004 INFO:teuthology.orchestra.run.smithi080.stderr: 94.3% -- replaced with /var/log/ceph/97c626fc-da3b-11ee-95c6-87774f69a715/ceph-osd.1.log.gz 2024-03-04T15:43:20.006 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-03-04T15:43:20.006 INFO:teuthology.orchestra.run.smithi080.stderr:real 0m0.331s 2024-03-04T15:43:20.006 INFO:teuthology.orchestra.run.smithi080.stderr:user 0m0.408s 2024-03-04T15:43:20.007 INFO:teuthology.orchestra.run.smithi080.stderr:sys 0m0.035s 2024-03-04T15:43:20.007 INFO:tasks.cephadm:Archiving logs... 2024-03-04T15:43:20.007 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/log/ceph to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216/remote/smithi005/log 2024-03-04T15:43:20.008 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T15:43:20.156 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/log/ceph to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216/remote/smithi080/log 2024-03-04T15:43:20.157 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T15:43:20.251 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/log/ceph to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216/remote/smithi087/log 2024-03-04T15:43:20.255 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T15:43:20.342 INFO:tasks.cephadm:Removing cluster... 2024-03-04T15:43:20.343 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 --force 2024-03-04T15:43:20.527 INFO:teuthology.orchestra.run.smithi005.stdout:Deleting cluster with fsid: 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:43:20.646 DEBUG:teuthology.orchestra.run.smithi080:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 --force 2024-03-04T15:43:20.832 INFO:teuthology.orchestra.run.smithi080.stdout:Deleting cluster with fsid: 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:43:20.945 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 97c626fc-da3b-11ee-95c6-87774f69a715 --force 2024-03-04T15:43:21.128 INFO:teuthology.orchestra.run.smithi087.stdout:Deleting cluster with fsid: 97c626fc-da3b-11ee-95c6-87774f69a715 2024-03-04T15:43:21.242 INFO:tasks.cephadm:Removing cephadm ... 2024-03-04T15:43:21.242 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T15:43:21.260 DEBUG:teuthology.orchestra.run.smithi080:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T15:43:21.275 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T15:43:21.290 INFO:tasks.cephadm:Teardown complete 2024-03-04T15:43:21.290 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-04T15:43:21.300 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi005:/dev/vg_nvme/lv_1... 2024-03-04T15:43:21.300 DEBUG:teuthology.orchestra.run.smithi005:> sudo nvme disconnect -n lv_1 2024-03-04T15:43:21.441 INFO:teuthology.orchestra.run.smithi005.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-04T15:43:21.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:21.443 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi005:/dev/vg_nvme/lv_2... 2024-03-04T15:43:21.443 DEBUG:teuthology.orchestra.run.smithi005:> sudo nvme disconnect -n lv_2 2024-03-04T15:43:21.586 INFO:teuthology.orchestra.run.smithi005.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-04T15:43:21.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:21.588 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi005:/dev/vg_nvme/lv_3... 2024-03-04T15:43:21.588 DEBUG:teuthology.orchestra.run.smithi005:> sudo nvme disconnect -n lv_3 2024-03-04T15:43:21.747 INFO:teuthology.orchestra.run.smithi005.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-04T15:43:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:21.749 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi005:/dev/vg_nvme/lv_4... 2024-03-04T15:43:21.749 DEBUG:teuthology.orchestra.run.smithi005:> sudo nvme disconnect -n lv_4 2024-03-04T15:43:21.880 INFO:teuthology.orchestra.run.smithi005.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-04T15:43:21.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:21.882 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-04T15:43:21.882 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/scratch_devs 2024-03-04T15:43:21.912 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi080:/dev/vg_nvme/lv_1... 2024-03-04T15:43:21.913 DEBUG:teuthology.orchestra.run.smithi080:> sudo nvme disconnect -n lv_1 2024-03-04T15:43:22.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:22.073 INFO:teuthology.orchestra.run.smithi080.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-04T15:43:22.073 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi080:/dev/vg_nvme/lv_2... 2024-03-04T15:43:22.073 DEBUG:teuthology.orchestra.run.smithi080:> sudo nvme disconnect -n lv_2 2024-03-04T15:43:22.224 INFO:teuthology.orchestra.run.smithi080.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-04T15:43:22.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:22.226 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi080:/dev/vg_nvme/lv_3... 2024-03-04T15:43:22.226 DEBUG:teuthology.orchestra.run.smithi080:> sudo nvme disconnect -n lv_3 2024-03-04T15:43:22.369 INFO:teuthology.orchestra.run.smithi080.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-04T15:43:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:22.370 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi080:/dev/vg_nvme/lv_4... 2024-03-04T15:43:22.371 DEBUG:teuthology.orchestra.run.smithi080:> sudo nvme disconnect -n lv_4 2024-03-04T15:43:22.519 INFO:teuthology.orchestra.run.smithi080.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-04T15:43:22.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:22.548 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-04T15:43:22.548 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/scratch_devs 2024-03-04T15:43:22.579 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi087:/dev/vg_nvme/lv_1... 2024-03-04T15:43:22.580 DEBUG:teuthology.orchestra.run.smithi087:> sudo nvme disconnect -n lv_1 2024-03-04T15:43:22.742 INFO:teuthology.orchestra.run.smithi087.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-04T15:43:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:22.743 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi087:/dev/vg_nvme/lv_2... 2024-03-04T15:43:22.744 DEBUG:teuthology.orchestra.run.smithi087:> sudo nvme disconnect -n lv_2 2024-03-04T15:43:22.898 INFO:teuthology.orchestra.run.smithi087.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-04T15:43:22.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:22.899 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi087:/dev/vg_nvme/lv_3... 2024-03-04T15:43:22.900 DEBUG:teuthology.orchestra.run.smithi087:> sudo nvme disconnect -n lv_3 2024-03-04T15:43:23.043 INFO:teuthology.orchestra.run.smithi087.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-04T15:43:23.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:23.044 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi087:/dev/vg_nvme/lv_4... 2024-03-04T15:43:23.044 DEBUG:teuthology.orchestra.run.smithi087:> sudo nvme disconnect -n lv_4 2024-03-04T15:43:23.190 INFO:teuthology.orchestra.run.smithi087.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-04T15:43:23.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:23.192 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-03-04T15:43:23.192 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/scratch_devs 2024-03-04T15:43:23.223 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-04T15:43:23.235 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-04T15:43:23.235 DEBUG:teuthology.orchestra.run.smithi005:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:43:23.238 DEBUG:teuthology.orchestra.run.smithi080:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:43:23.240 DEBUG:teuthology.orchestra.run.smithi087:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:43:23.252 INFO:teuthology.orchestra.run.smithi005.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:43:23.254 INFO:teuthology.orchestra.run.smithi080.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:43:23.257 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:43:23.257 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2024-03-04T15:43:23.257 INFO:teuthology.orchestra.run.smithi005.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 -3314us[-3314us] +/- 102ms 2024-03-04T15:43:23.257 INFO:teuthology.orchestra.run.smithi005.stdout:^* hv02.front.sepia.ceph.com 3 6 377 39 -844us[ -839us] +/- 71ms 2024-03-04T15:43:23.257 INFO:teuthology.orchestra.run.smithi005.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +3546us[+3546us] +/- 85ms 2024-03-04T15:43:23.257 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:43:23.258 INFO:teuthology.orchestra.run.smithi080.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:43:23.259 INFO:teuthology.orchestra.run.smithi080.stdout:=============================================================================== 2024-03-04T15:43:23.259 INFO:teuthology.orchestra.run.smithi080.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 -3193us[-3236us] +/- 102ms 2024-03-04T15:43:23.259 INFO:teuthology.orchestra.run.smithi080.stdout:^* hv02.front.sepia.ceph.com 3 6 377 35 -717us[ -760us] +/- 72ms 2024-03-04T15:43:23.259 INFO:teuthology.orchestra.run.smithi080.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 +3658us[+3616us] +/- 85ms 2024-03-04T15:43:23.259 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:43:23.279 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:43:23.284 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:43:23.284 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-03-04T15:43:23.284 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 -3219us[-3219us] +/- 102ms 2024-03-04T15:43:23.284 INFO:teuthology.orchestra.run.smithi087.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -749us[ -779us] +/- 72ms 2024-03-04T15:43:23.284 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +3623us[+3623us] +/- 85ms 2024-03-04T15:43:23.284 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:43:23.285 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-04T15:43:23.294 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-04T15:43:23.295 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-04T15:43:23.304 DEBUG:teuthology.orchestra.run.smithi005:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T15:43:23.385 DEBUG:teuthology.orchestra.run.smithi080:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T15:43:23.460 DEBUG:teuthology.orchestra.run.smithi087:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T15:43:23.536 DEBUG:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:43:23.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:23.568 DEBUG:teuthology.orchestra.run.smithi080:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:43:23.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:23.601 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:43:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:23.634 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-04T15:43:23.644 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-04T15:43:23.652 INFO:teuthology.task.internal:Duration was 1307.399350 seconds 2024-03-04T15:43:23.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-04T15:43:23.661 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-04T15:43:23.661 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T15:43:23.664 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T15:43:23.667 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T15:43:23.707 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:43:23.710 INFO:teuthology.orchestra.run.smithi080.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:43:23.718 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:43:24.219 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-04T15:43:24.219 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi005.front.sepia.ceph.com 2024-03-04T15:43:24.220 DEBUG:teuthology.orchestra.run.smithi005:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T15:43:24.241 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi080.front.sepia.ceph.com 2024-03-04T15:43:24.241 DEBUG:teuthology.orchestra.run.smithi080:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T15:43:24.270 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com 2024-03-04T15:43:24.270 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T15:43:24.294 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-04T15:43:24.294 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T15:43:24.297 DEBUG:teuthology.orchestra.run.smithi080:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T15:43:24.312 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T15:43:24.365 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-04T15:43:24.365 DEBUG:teuthology.orchestra.run.smithi005:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T15:43:24.368 DEBUG:teuthology.orchestra.run.smithi080:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T15:43:24.384 DEBUG:teuthology.orchestra.run.smithi087:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T15:43:24.568 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-04T15:43:24.578 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-04T15:43:24.578 DEBUG:teuthology.orchestra.run.smithi005:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T15:43:24.605 DEBUG:teuthology.orchestra.run.smithi080:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T15:43:24.632 DEBUG:teuthology.orchestra.run.smithi087:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T15:43:24.659 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-04T15:43:24.670 DEBUG:teuthology.orchestra.run.smithi005:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T15:43:24.672 DEBUG:teuthology.orchestra.run.smithi080:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T15:43:24.675 DEBUG:teuthology.orchestra.run.smithi087:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T15:43:24.694 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = core 2024-03-04T15:43:24.698 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = core 2024-03-04T15:43:24.725 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = core 2024-03-04T15:43:24.748 DEBUG:teuthology.orchestra.run.smithi005:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T15:43:24.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:24.772 DEBUG:teuthology.orchestra.run.smithi080:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T15:43:24.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:24.786 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T15:43:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:43:24.803 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-04T15:43:24.813 INFO:teuthology.task.internal:Transferring archived files... 2024-03-04T15:43:24.814 DEBUG:teuthology.misc:Transferring archived files from smithi005:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216/remote/smithi005 2024-03-04T15:43:24.814 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T15:43:24.853 DEBUG:teuthology.misc:Transferring archived files from smithi080:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216/remote/smithi080 2024-03-04T15:43:24.853 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T15:43:24.890 DEBUG:teuthology.misc:Transferring archived files from smithi087:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216/remote/smithi087 2024-03-04T15:43:24.891 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T15:43:24.929 INFO:teuthology.task.internal:Removing archive directory... 2024-03-04T15:43:24.929 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T15:43:24.932 DEBUG:teuthology.orchestra.run.smithi080:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T15:43:24.934 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T15:43:24.978 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-04T15:43:24.990 INFO:teuthology.task.internal:Not uploading archives. 2024-03-04T15:43:24.990 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-04T15:43:24.999 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-04T15:43:24.999 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T15:43:25.001 DEBUG:teuthology.orchestra.run.smithi080:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T15:43:25.003 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T15:43:25.013 INFO:teuthology.orchestra.run.smithi005.stdout: 265921 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 15:43 /home/ubuntu/cephtest 2024-03-04T15:43:25.015 INFO:teuthology.orchestra.run.smithi080.stdout: 265459 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 15:43 /home/ubuntu/cephtest 2024-03-04T15:43:25.032 INFO:teuthology.orchestra.run.smithi087.stdout: 265690 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 15:43 /home/ubuntu/cephtest 2024-03-04T15:43:25.033 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-04T15:43:25.063 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-04T15:43:25.159 INFO:teuthology.nuke:Checking targets against current locks 2024-03-04T15:43:25.195 DEBUG:teuthology.nuke:shortname: smithi005 2024-03-04T15:43:25.195 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T15:43:25.240 DEBUG:teuthology.nuke:shortname: smithi080 2024-03-04T15:43:25.240 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T15:43:25.260 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:09:16.519420', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAEwl8LNvr8Q+ASGvGG4IArMD4y7zEJYEF3RiJCGvEEMSIZC4whN+b1tyKctc8owViKgIZEVu21QpHX7U07jn7Q='} 2024-03-04T15:43:25.281 DEBUG:teuthology.nuke:shortname: smithi087 2024-03-04T15:43:25.282 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T15:43:25.293 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi080.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:09:16.517521', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGKJoDJ0ZThG8/SoNJ8S1EiuczPEDkJyIH/ICdHs9wI/Jkz5UvvKzjew480GSGpdPzp1tpLKdk/qLfYuo/6yDuc='} 2024-03-04T15:43:25.345 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581216', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:09:16.521161', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHg2B34znpP0NOSpnSvpFgaQ5yUZInWSIRBio/OYMkB7Qb+6VQzc37lG9FHlYLhXGyfULbVP9HeVuss9lnsj2S0='} 2024-03-04T15:43:25.350 INFO:teuthology.orchestra.console.smithi005:Power off 2024-03-04T15:43:25.351 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T15:43:25.412 INFO:teuthology.orchestra.console.smithi080:Power off 2024-03-04T15:43:25.413 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T15:43:25.420 DEBUG:teuthology.orchestra.console.smithi005:power off output: Chassis Power Control: Down/Off 2024-03-04T15:43:25.420 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:43:25.457 INFO:teuthology.orchestra.console.smithi087:Power off 2024-03-04T15:43:25.457 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T15:43:25.466 DEBUG:teuthology.orchestra.console.smithi080:power off output: Chassis Power Control: Down/Off 2024-03-04T15:43:25.467 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:43:25.476 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is on 2024-03-04T15:43:25.506 DEBUG:teuthology.orchestra.console.smithi087:power off output: Chassis Power Control: Down/Off 2024-03-04T15:43:25.506 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:43:25.517 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is on 2024-03-04T15:43:25.554 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2024-03-04T15:43:29.477 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:43:29.518 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:43:29.555 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:43:29.588 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is on 2024-03-04T15:43:29.628 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is on 2024-03-04T15:43:29.666 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2024-03-04T15:43:33.592 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:43:33.629 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:43:33.667 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:43:33.701 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is off 2024-03-04T15:43:33.738 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is off 2024-03-04T15:43:33.778 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is off 2024-03-04T15:43:33.803 INFO:teuthology.orchestra.console.smithi005:Power off completed 2024-03-04T15:43:33.839 INFO:teuthology.orchestra.console.smithi080:Power off completed 2024-03-04T15:43:33.879 INFO:teuthology.orchestra.console.smithi087:Power off completed 2024-03-04T15:43:33.983 INFO:teuthology.run:Summary data: description: orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} duration: 1307.3993504047394 failure_reason: '"2024-03-04T15:38:57.904963+0000 mon.a (mon.0) 248 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_tonay@teuthology success: false 2024-03-04T15:43:33.983 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T15:43:34.092 INFO:teuthology.run:FAIL